Controlling vehicles in response to windows

ABSTRACT

Systems, methods and computer readable media for controlling vehicles in response to windows are provided. One or more images captured using one or more image sensors from an environment of a vehicle may be obtained. The one or more images may be analyzed to detect a first window in the environment. The vehicle may be navigated to a stopping position, where in the stopping position a window of the vehicle may be positioned at a selected position with respect to the first window.

CROSS REFERENCES TO RELATED APPLICATIONS

This application claims priority under 35 U.S.C. § 119 to U.S.Provisional Patent Application No. 62/989,847, filed on Mar. 15, 2020.The entire contents of all of the above-identified applications areherein incorporated by reference.

BACKGROUND Technological Field

The disclosed embodiments generally relate to systems, methods andcomputer readable media for controlling vehicles and vehicle relatedsystems. More specifically, the disclosed embodiments relate to systems,methods and computer readable media for controlling vehicles and vehiclerelated systems in response to windows.

Background Information

Usage of vehicles is common and key to many everyday activities.

Audio and image sensors, as well as other sensors, are now part ofnumerous devices, from mobile phones to vehicles, and the availabilityof audio data and image data, as well as other information produced bythese devices, is increasing.

SUMMARY

In some embodiments, systems, methods and computer readable media forcontrolling vehicles and vehicle related systems are provided.

In some embodiments, systems, methods and computer readable media forcontrolling vehicles in response to cranes are provided. For example,one or more images captured using one or more image sensors from anenvironment of a first vehicle may be obtained. The one or more imagesmay be analyzed to detect a second vehicle. The one or more images maybe analyzed to determine that the second vehicle is connected to acrane. The one or more images may be analyzed to determine a state ofthe crane. In response to a first determined state of the crane, thefirst vehicle may be caused to initiate an action responding to thesecond vehicle, and in response to a second determined state of thecrane, causing the first vehicle to initiate the action may be withheld.

In some embodiments, systems, methods and computer readable media forcontrolling vehicles in response to lifts are provided. For example, oneor more images captured using one or more image sensors from anenvironment of a first vehicle may be obtained. The one or more imagesmay be analyzed to detect a second vehicle. The one or more images maybe analyzed to determine that the second vehicle is connected to a lift.The one or more images may be analyzed to determine a state of the lift.In response to a first determined state of the lift, the first vehiclemay be caused to initiate an action responding to the second vehicle,and in response to a second determined state of the lift, causing thefirst vehicle to initiate the action may be withheld.

In some embodiments, systems, methods and computer readable media forcontrolling vehicles in response to outriggers are provided. Forexample, one or more images captured using one or more image sensorsfrom an environment of a first vehicle may be obtained. The one or moreimages may be analyzed to detect a second vehicle. The one or moreimages may be analyzed to determine that the second vehicle is connectedto an outrigger. The one or more images may be analyzed to determine astate of the outrigger. In response to a first determined state of theoutrigger, the first vehicle may be caused to initiate an actionresponding to the second vehicle, and in response to a second determinedstate of the outrigger, causing the first vehicle to initiate the actionmay be withheld.

In some embodiments, systems, methods and computer readable media forcontrolling vehicles in response to pumps are provided. For example, oneor more images captured using one or more image sensors from anenvironment of a first vehicle may be obtained. The one or more imagesmay be analyzed to detect a second vehicle. The one or more images maybe analyzed to determine that the second vehicle is connected to a pump.The one or more images may be analyzed to determine a state of the pump.In response to a first determined state of the pump, the first vehiclemay be caused to initiate an action responding to the second vehicle,and in response to a second determined state of the pump, causing thefirst vehicle to initiate the action may be withheld.

In some embodiments, systems, methods and computer readable media forcontrolling vehicles in response to pipes are provided. For example, oneor more images captured using one or more image sensors from anenvironment of a first vehicle may be obtained. The one or more imagesmay be analyzed to detect a second vehicle. The one or more images maybe analyzed to determine that the second vehicle is connected to a pipe.The one or more images may be analyzed to determine a state of the pipe.In response to a first determined state of the pipe, the first vehiclemay be caused to initiate an action responding to the second vehicle,and in response to a second determined state of the pipe, causing thefirst vehicle to initiate the action may be withheld.

In some embodiments, systems, methods and computer readable media forcontrolling vehicles in response to objects are provided. For example,one or more images captured using one or more image sensors from anenvironment of a first vehicle may be obtained. The one or more imagesmay be analyzed to detect an object. It may be determined whether theobject is carried by a second vehicle. In response to a determinationthat the object is not carried by a second vehicle, the first vehiclemay be caused to initiate an action responding to the object, and inresponse to a determination that the object is carried by a secondvehicle, causing the first vehicle to initiate the action may bewithheld.

In some embodiments, systems, methods and computer readable media forcontrolling vehicles based on doors of other vehicles are provided. Forexample, one or more images captured using one or more image sensorsfrom an environment of a first vehicle may be obtained. The one or moreimages may be analyzed to detect a second vehicle. The one or moreimages may be analyzed to determine a state of a door of the secondvehicle. In response to a first determined state of the door, the firstvehicle may be caused to initiate an action responding to the secondvehicle, and in response to a second determined state of the door,causing the first vehicle to initiate the action may be withheld.

In some embodiments, systems, methods and computer readable media forcontrolling vehicles based on users of other vehicles are provided. Forexample, one or more images captured using one or more image sensorsfrom an environment of a first vehicle may be obtained. The one or moreimages may be analyzed to detect a second vehicle. The one or moreimages may be analyzed to determine a state of a user associated withthe second vehicle. In response to a first determined state of the user,the first vehicle may be caused to initiate an action responding to thesecond vehicle, and in response to a second determined state of theuser, causing the first vehicle to initiate the action may be withheld.

In some embodiments, systems, methods and computer readable media forcontrolling vehicles based on loading and unloading events are provided.For example, one or more images captured using one or more image sensorsfrom an environment of a first vehicle may be obtained. The one or moreimages may be analyzed to detect a second vehicle. At least one of acargo loading event associated with the second vehicle and a cargounloading event associated with the second vehicle may be determined. Inresponse to the determined at least one of cargo loading eventassociated with the second vehicle and cargo unloading event associatedwith the second vehicle, the first vehicle may be caused to initiate anaction responding to the second vehicle.

In some embodiments, systems, methods and computer readable media forcontrolling vehicles in response to street cleaning vehicles areprovided. For example, one or more images captured using one or moreimage sensors from an environment of a first vehicle may be obtained.The one or more images may be analyzed to detect a second vehicle. Theone or more images may be analyzed to determine that the second vehicleis a street cleaning vehicle. In response to the determination that thesecond vehicle is a street cleaning vehicle, the first vehicle may becaused to initiate an action responding to the second vehicle.

In some embodiments, systems, methods and computer readable media forcontrolling vehicles based on hoods of other vehicles are provided. Forexample, one or more images captured using one or more image sensorsfrom an environment of a first vehicle may be obtained. The one or moreimages may be analyzed to detect a second vehicle. The one or moreimages may be analyzed to determine a state of a hood of the secondvehicle. In response to a first determined state of the hood, the firstvehicle may be caused to initiate an action responding to the secondvehicle, and in response to a second determined state of the hood,causing the first vehicle to initiate the action may be withheld.

In some embodiments, systems, methods and computer readable media forcontrolling vehicles based on trunk lids of other vehicles are provided.For example, one or more images captured using one or more image sensorsfrom an environment of a first vehicle may be obtained. The one or moreimages may be analyzed to detect a second vehicle. The one or moreimages may be analyzed to determine a state of a trunk lid of the secondvehicle. In response to a first determined state of the trunk lid, thefirst vehicle may be caused to initiate an action responding to thesecond vehicle, and in response to a second determined state of thetrunk lid, causing the first vehicle to initiate the action may bewithheld.

In some embodiments, systems, methods and computer readable media forcontrolling vehicles in response to smoke are provided. For example, oneor more images captured using one or more image sensors from anenvironment of a first vehicle may be obtained. The one or more imagesmay be analyzed to detect a second vehicle. The one or more images maybe analyzed to determine that the second vehicle emits smoke not throughan exhaust system. In response to the determination that the secondvehicle emits smoke not through an exhaust system, the first vehicle maybe caused to initiate an action responding to the second vehicle.

In some embodiments, systems, methods and computer readable media forcontrolling vehicles in response to two-wheeler vehicles are provided.For example, one or more images captured using one or more image sensorsfrom an environment of a first vehicle may be obtained. The one or moreimages may be analyzed to detect a two-wheeler vehicle. The one or moreimages may be analyzed to determine whether at least one rider rides thetwo-wheeler vehicle, and in response to a determination that no riderrides the two-wheeler vehicle, the first vehicle may be caused toinitiate an action responding to the two-wheeler vehicle.

In some embodiments, systems, methods and computer readable media forcontrolling vehicles in response to winter service vehicles areprovided. For example, one or more images captured using one or moreimage sensors from an environment of a first vehicle may be obtained.The one or more images may be analyzed to detect a second vehicle. Theone or more images may be analyzed to determine that the second vehicleis a winter service vehicle. In response to the determination that thesecond vehicle is a winter service vehicle, the first vehicle may becaused to initiate an action responding to the second vehicle.

In some embodiments, systems, methods and computer readable media forcontrolling vehicles in response to waste collection vehicles areprovided. For example, one or more images captured using one or moreimage sensors from an environment of a first vehicle may be obtained.The one or more images may be analyzed to detect a second vehicle. Theone or more images may be analyzed to determine that the second vehicleis a waste collection vehicle. In response to the determination that thesecond vehicle is a waste collection vehicle, the first vehicle may becaused to initiate an action responding to the second vehicle.

In some embodiments, systems, methods and computer readable media forcontrolling vehicles in response to windows are provided. For example,one or more images captured using one or more image sensors from anenvironment of a vehicle may be obtained. The one or more images may beanalyzed to detect a first window in the environment. The vehicle may benavigated to a stopping position, where in the stopping position awindow of the vehicle may be positioned at a selected position withrespect to the first window.

BRIEF DESCRIPTION OF THE DRAWINGS

FIGS. 1A and 1B are block diagrams illustrating some possibleimplementations of a communicating system.

FIGS. 2A and 2B are block diagrams illustrating some possibleimplementations of an apparatus.

FIG. 3 is a block diagram illustrating a possible implementation of aserver.

FIGS. 4A and 4B are block diagrams illustrating some possibleimplementations of a cloud platform.

FIG. 5 is a block diagram illustrating a possible implementation of acomputational node.

FIG. 6 is a schematic illustration of example images of trucksconsistent with an embodiment of the present disclosure.

FIG. 7 illustrates an example of a method for controlling vehicles inresponse to cranes.

FIG. 8 illustrates an example of a method for controlling vehicles inresponse to lifts.

FIG. 9 illustrates an example of a method for controlling vehicles inresponse to outriggers.

FIG. 10 illustrates an example of a method for controlling vehicles inresponse to pumps.

FIG. 11 illustrates an example of a method for controlling vehicles inresponse to pipes.

FIG. 12 illustrates an example of a method for controlling vehicles inresponse to objects.

FIG. 13 illustrates an example of a method for controlling vehiclesbased on doors of other vehicles.

FIG. 14 illustrates an example of a method for controlling vehiclesbased on users of other vehicles.

FIG. 15 illustrates an example of a method for controlling vehiclesbased on loading and/or unloading events.

FIG. 16 illustrates an example of a method for controlling vehicles inresponse to street cleaning vehicles.

FIG. 17 illustrates an example of a method for controlling vehiclesbased on hoods of other vehicles.

FIG. 18 illustrates an example of a method for controlling vehiclesbased on trunk lids of other vehicles.

FIG. 19 illustrates an example of a method 1900 for controlling vehiclesin response to smoke.

FIG. 20 illustrates an example of a method for controlling vehicles inresponse to two-wheeler vehicles.

FIG. 21 illustrates an example of a method for controlling vehicles inresponse to winter service vehicles.

FIG. 22 illustrates an example of a method for controlling vehicles inresponse to waste collection vehicles.

FIG. 23 illustrates an example of a method for controlling vehicles inresponse to windows.

DESCRIPTION

Unless specifically stated otherwise, as apparent from the followingdiscussions, it is appreciated that throughout the specificationdiscussions utilizing terms such as “processing”, “calculating”,“computing”, “determining”, “generating”, “setting”, “configuring”,“selecting”, “defining”, “applying”, “obtaining”, “monitoring”,“providing”, “identifying”, “segmenting”, “classifying”, “analyzing”,“associating”, “extracting”, “storing”, “receiving”, “transmitting”, orthe like, include action and/or processes of a computer that manipulateand/or transform data into other data, said data represented as physicalquantities, for example such as electronic quantities, and/or said datarepresenting the physical objects. The terms “computer”, “processor”,“controller”, “processing unit”, “computing unit”, and “processingmodule” should be expansively construed to cover any kind of electronicdevice, component or unit with data processing capabilities, including,by way of non-limiting example, a personal computer, a wearablecomputer, a tablet, a smartphone, a server, a computing system, a cloudcomputing platform, a communication device, a processor (for example,digital signal processor (DSP), an image signal processor (ISR), amicrocontroller, a field programmable gate array (FPGA), an applicationspecific integrated circuit (ASIC), a central processing unit (CPA), agraphics processing unit (GPU), a visual processing unit (VPU), and soon), possibly with embedded memory, a single core processor, a multicore processor, a core within a processor, any other electroniccomputing device, or any combination of the above.

The operations in accordance with the teachings herein may be performedby a computer specially constructed or programmed to perform thedescribed functions.

As used herein, the phrase “for example,” “such as”, “for instance” andvariants thereof describe non-limiting embodiments of the presentlydisclosed subject matter. Reference in the specification to “one case”,“some cases”, “other cases” or variants thereof means that a particularfeature, structure or characteristic described in connection with theembodiment(s) may be included in at least one embodiment of thepresently disclosed subject matter. Thus the appearance of the phrase“one case”, “some cases”, “other cases” or variants thereof does notnecessarily refer to the same embodiment(s). As used herein, the term“and/or” includes any and all combinations of one or more of theassociated listed items.

It is appreciated that certain features of the presently disclosedsubject matter, which are, for clarity, described in the context ofseparate embodiments, may also be provided in combination in a singleembodiment. Conversely, various features of the presently disclosedsubject matter, which are, for brevity, described in the context of asingle embodiment, may also be provided separately or in any suitablesub-combination.

The term “image sensor” is recognized by those skilled in the art andrefers to any device configured to capture images, a sequence of images,videos, and so forth. This includes sensors that convert optical inputinto images, where optical input can be visible light (like in acamera), radio waves, microwaves, terahertz waves, ultraviolet light,infrared light, x-rays, gamma rays, and/or any other light spectrum.This also includes both 2D and 3D sensors. Examples of image sensortechnologies may include: CCD, CMOS, NMOS, and so forth. 3D sensors maybe implemented using different technologies, including: stereo camera,active stereo camera, time of flight camera, structured light camera,radar, range image camera, and so forth.

In embodiments of the presently disclosed subject matter, one or morestages illustrated in the figures may be executed in a different orderand/or one or more groups of stages may be executed simultaneously andvice versa. The figures illustrate a general schematic of the systemarchitecture in accordance embodiments of the presently disclosedsubject matter. Each module in the figures can be made up of anycombination of software, hardware and/or firmware that performs thefunctions as defined and explained herein. The modules in the figuresmay be centralized in one location or dispersed over more than onelocation.

It should be noted that some examples of the presently disclosed subjectmatter are not limited in application to the details of construction andthe arrangement of the components set forth in the following descriptionor illustrated in the drawings. The invention can be capable of otherembodiments or of being practiced or carried out in various ways. Also,it is to be understood that the phraseology and terminology employedherein is for the purpose of description and should not be regarded aslimiting.

In this document, an element of a drawing that is not described withinthe scope of the drawing and is labeled with a numeral that has beendescribed in a previous drawing may have the same use and description asin the previous drawings.

The drawings in this document may not be to any scale. Different figuresmay use different scales and different scales can be used even withinthe same drawing, for example different scales for different views ofthe same object or different scales for the two adjacent objects.

FIG. 1A is a block diagram illustrating a possible implementation of acommunicating system. In this example, apparatuses 200 a and 200 b maycommunicate with server 300 a, with server 300 b, with cloud platform400, with each other, and so forth. Possible implementations ofapparatuses 200 a and 200 b may include apparatus 200 as described inFIGS. 2A and 2B. Possible implementations of servers 300 a and 300 b mayinclude server 300 as described in FIG. 3 . Some possibleimplementations of cloud platform 400 are described in FIGS. 4A, 4B and5 . In this example apparatuses 200 a and 200 b may communicate directlywith mobile phone 111, tablet 112, and personal computer (PC) 113.Apparatuses 200 a and 200 b may communicate with local router 120directly, and/or through at least one of mobile phone 111, tablet 112,and personal computer (PC) 113. In this example, local router 120 may beconnected with a communication network 130. Examples of communicationnetwork 130 may include the Internet, phone networks, cellular networks,satellite communication networks, private communication networks,virtual private networks (VPN), and so forth. Apparatuses 200 a and 200b may connect to communication network 130 through local router 120and/or directly. Apparatuses 200 a and 200 b may communicate with otherdevices, such as servers 300 a, server 300 b, cloud platform 400, remotestorage 140 and network attached storage (NAS) 150, throughcommunication network 130 and/or directly.

FIG. 1B is a block diagram illustrating a possible implementation of acommunicating system. In this example, apparatuses 200 a, 200 b and 200c may communicate with cloud platform 400 and/or with each other throughcommunication network 130. Possible implementations of apparatuses 200a, 200 b and 200 c may include apparatus 200 as described in FIGS. 2Aand 2B. Some possible implementations of cloud platform 400 aredescribed in FIGS. 4A, 4B and 5 .

FIGS. 1A and 1B illustrate some possible implementations of acommunication system. In some embodiments, other communication systemsthat enable communication between apparatus 200 and server 300 may beused. In some embodiments, other communication systems that enablecommunication between apparatus 200 and cloud platform 400 may be used.In some embodiments, other communication systems that enablecommunication among a plurality of apparatuses 200 may be used.

FIG. 2A is a block diagram illustrating a possible implementation ofapparatus 200. In this example, apparatus 200 may comprise: one or morememory units 210, one or more processing units 220, and one or moreimage sensors 260. In some implementations, apparatus 200 may compriseadditional components, while some components listed above may beexcluded.

FIG. 2B is a block diagram illustrating a possible implementation ofapparatus 200. In this example, apparatus 200 may comprise: one or morememory units 210, one or more processing units 220, one or morecommunication modules 230, one or more power sources 240, one or moreaudio sensors 250, one or more image sensors 260, one or more lightsources 265, one or more motion sensors 270, and one or more positioningsensors 275. In some implementations, apparatus 200 may compriseadditional components, while some components listed above may beexcluded. For example, in some implementations apparatus 200 may alsocomprise at least one of the following: one or more barometers; one ormore user input devices; one or more output devices; and so forth. Inanother example, in some implementations at least one of the followingmay be excluded from apparatus 200: memory units 210, communicationmodules 230, power sources 240, audio sensors 250, image sensors 260,light sources 265, motion sensors 270, and positioning sensors 275.

In some embodiments, one or more power sources 240 may be configured to:power apparatus 200; power server 300; power cloud platform 400; and/orpower computational node 500. Possible implementation examples of powersources 240 may include: one or more electric batteries; one or morecapacitors; one or more connections to external power sources; one ormore power convertors; any combination of the above; and so forth.

In some embodiments, the one or more processing units 220 may beconfigured to execute software programs. For example, processing units220 may be configured to execute software programs stored on the memoryunits 210. In some cases, the executed software programs may storeinformation in memory units 210. In some cases, the executed softwareprograms may retrieve information from the memory units 210. Possibleimplementation examples of the processing units 220 may include: one ormore single core processors, one or more multicore processors; one ormore controllers; one or more application processors; one or more systemon a chip processors; one or more central processing units; one or moregraphical processing units; one or more neural processing units; anycombination of the above; and so forth.

In some embodiments, the one or more communication modules 230 may beconfigured to receive and transmit information. For example, controlsignals may be transmitted and/or received through communication modules230. In another example, information received though communicationmodules 230 may be stored in memory units 210. In an additional example,information retrieved from memory units 210 may be transmitted usingcommunication modules 230. In another example, input data may betransmitted and/or received using communication modules 230. Examples ofsuch input data may include: input data inputted by a user using userinput devices; information captured using one or more sensors; and soforth. Examples of such sensors may include: audio sensors 250; imagesensors 260; motion sensors 270; positioning sensors 275; chemicalsensors; temperature sensors; barometers; and so forth.

In some embodiments, the one or more audio sensors 250 may be configuredto capture audio by converting sounds to digital information. Somenon-limiting examples of audio sensors 250 may include: microphones,unidirectional microphones, bidirectional microphones, cardioidmicrophones, omnidirectional microphones, onboard microphones, wiredmicrophones, wireless microphones, any combination of the above, and soforth. In some examples, the captured audio may be stored in memoryunits 210. In some additional examples, the captured audio may betransmitted using communication modules 230, for example to othercomputerized devices, such as server 300, cloud platform 400,computational node 500, and so forth. In some examples, processing units220 may control the above processes. For example, processing units 220may control at least one of: capturing of the audio; storing thecaptured audio; transmitting of the captured audio; and so forth. Insome cases, the captured audio may be processed by processing units 220.For example, the captured audio may be compressed by processing units220; possibly followed: by storing the compressed captured audio inmemory units 210; by transmitted the compressed captured audio usingcommunication modules 230; and so forth. In another example, thecaptured audio may be processed using speech recognition algorithms. Inanother example, the captured audio may be processed using speakerrecognition algorithms.

In some embodiments, the one or more image sensors 260 may be configuredto capture visual information by converting light to: images; sequenceof images; videos; 3D images; sequence of 3D images; 3D videos; and soforth. In some examples, the captured visual information may be storedin memory units 210. In some additional examples, the captured visualinformation may be transmitted using communication modules 230, forexample to other computerized devices, such as server 300, cloudplatform 400, computational node 500, and so forth. In some examples,processing units 220 may control the above processes. For example,processing units 220 may control at least one of: capturing of thevisual information; storing the captured visual information;transmitting of the captured visual information; and so forth. In somecases, the captured visual information may be processed by processingunits 220. For example, the captured visual information may becompressed by processing units 220; possibly followed: by storing thecompressed captured visual information in memory units 210; bytransmitted the compressed captured visual information usingcommunication modules 230; and so forth. In another example, thecaptured visual information may be processed in order to: detectobjects, detect events, detect action, detect face, detect people,recognize person, and so forth.

In some embodiments, the one or more light sources 265 may be configuredto emit light, for example in order to enable better image capturing byimage sensors 260. In some examples, the emission of light may becoordinated with the capturing operation of image sensors 260. In someexamples, the emission of light may be continuous. In some examples, theemission of light may be performed at selected times. The emitted lightmay be visible light, infrared light, x-rays, gamma rays, and/or in anyother light spectrum. In some examples, image sensors 260 may capturelight emitted by light sources 265, for example in order to capture 3Dimages and/or 3D videos using active stereo method.

In some embodiments, the one or more motion sensors 270 may beconfigured to perform at least one of the following: detect motion ofobjects in the environment of apparatus 200; measure the velocity ofobjects in the environment of apparatus 200; measure the acceleration ofobjects in the environment of apparatus 200; detect motion of apparatus200; measure the velocity of apparatus 200; measure the acceleration ofapparatus 200; and so forth. In some implementations, the one or moremotion sensors 270 may comprise one or more accelerometers configured todetect changes in proper acceleration and/or to measure properacceleration of apparatus 200. In some implementations, the one or moremotion sensors 270 may comprise one or more gyroscopes configured todetect changes in the orientation of apparatus 200 and/or to measureinformation related to the orientation of apparatus 200. In someimplementations, motion sensors 270 may be implemented using imagesensors 260, for example by analyzing images captured by image sensors260 to perform at least one of the following tasks: track objects in theenvironment of apparatus 200; detect moving objects in the environmentof apparatus 200; measure the velocity of objects in the environment ofapparatus 200; measure the acceleration of objects in the environment ofapparatus 200; measure the velocity of apparatus 200, for example bycalculating the egomotion of image sensors 260; measure the accelerationof apparatus 200, for example by calculating the egomotion of imagesensors 260; and so forth. In some implementations, motion sensors 270may be implemented using image sensors 260 and light sources 265, forexample by implementing a LIDAR using image sensors 260 and lightsources 265. In some implementations, motion sensors 270 may beimplemented using one or more RADARs. In some examples, informationcaptured using motion sensors 270: may be stored in memory units 210,may be processed by processing units 220, may be transmitted and/orreceived using communication modules 230, and so forth.

In some embodiments, the one or more positioning sensors 275 may beconfigured to obtain positioning information of apparatus 200, to detectchanges in the position of apparatus 200, and/or to measure the positionof apparatus 200. In some examples, positioning sensors 275 may beimplemented using one of the following technologies: Global PositioningSystem (GPS), GLObal NAvigation Satellite System (GLONASS), Galileoglobal navigation system, BeiDou navigation system, other GlobalNavigation Satellite Systems (GNSS), Indian Regional NavigationSatellite System (IRNSS), Local Positioning Systems (LPS), Real-TimeLocation Systems (RTLS), Indoor Positioning System (IPS), Wi-Fi basedpositioning systems, cellular triangulation, and so forth. In someexamples, information captured using positioning sensors 275 may bestored in memory units 210, may be processed by processing units 220,may be transmitted and/or received using communication modules 230, andso forth.

In some embodiments, the one or more chemical sensors may be configuredto perform at least one of the following: measure chemical properties inthe environment of apparatus 200; measure changes in the chemicalproperties in the environment of apparatus 200; detect the present ofchemicals in the environment of apparatus 200; measure the concentrationof chemicals in the environment of apparatus 200. Examples of suchchemical properties may include: pH level, toxicity, temperature, and soforth. Examples of such chemicals may include: electrolytes, particularenzymes, particular hormones, particular proteins, smoke, carbondioxide, carbon monoxide, oxygen, ozone, hydrogen, hydrogen sulfide, andso forth. In some examples, information captured using chemical sensorsmay be stored in memory units 210, may be processed by processing units220, may be transmitted and/or received using communication modules 230,and so forth.

In some embodiments, the one or more temperature sensors may beconfigured to detect changes in the temperature of the environment ofapparatus 200 and/or to measure the temperature of the environment ofapparatus 200. In some examples, information captured using temperaturesensors may be stored in memory units 210, may be processed byprocessing units 220, may be transmitted and/or received usingcommunication modules 230, and so forth.

In some embodiments, the one or more barometers may be configured todetect changes in the atmospheric pressure in the environment ofapparatus 200 and/or to measure the atmospheric pressure in theenvironment of apparatus 200. In some examples, information capturedusing the barometers may be stored in memory units 210, may be processedby processing units 220, may be transmitted and/or received usingcommunication modules 230, and so forth.

In some embodiments, the one or more user input devices may beconfigured to allow one or more users to input information. In someexamples, user input devices may comprise at least one of the following:a keyboard, a mouse, a touch pad, a touch screen, a joystick, amicrophone, an image sensor, and so forth. In some examples, the userinput may be in the form of at least one of: text, sounds, speech, handgestures, body gestures, tactile information, and so forth. In someexamples, the user input may be stored in memory units 210, may beprocessed by processing units 220, may be transmitted and/or receivedusing communication modules 230, and so forth.

In some embodiments, the one or more user output devices may beconfigured to provide output information to one or more users. In someexamples, such output information may comprise of at least one of:notifications, feedbacks, reports, and so forth. In some examples, useroutput devices may comprise at least one of: one or more audio outputdevices; one or more textual output devices; one or more visual outputdevices; one or more tactile output devices; and so forth. In someexamples, the one or more audio output devices may be configured tooutput audio to a user, for example through: a headset, a set ofspeakers, and so forth. In some examples, the one or more visual outputdevices may be configured to output visual information to a user, forexample through: a display screen, an augmented reality display system,a printer, a LED indicator, and so forth. In some examples, the one ormore tactile output devices may be configured to output tactilefeedbacks to a user, for example through vibrations, through motions, byapplying forces, and so forth. In some examples, the output may beprovided: in real time, offline, automatically, upon request, and soforth. In some examples, the output information may be read from memoryunits 210, may be provided by a software executed by processing units220, may be transmitted and/or received using communication modules 230,and so forth.

FIG. 3 is a block diagram illustrating a possible implementation ofserver 300. In this example, server 300 may comprise: one or more memoryunits 210, one or more processing units 220, one or more communicationmodules 230, and one or more power sources 240. In some implementations,server 300 may comprise additional components, while some componentslisted above may be excluded. For example, in some implementationsserver 300 may also comprise at least one of the following: one or moreuser input devices; one or more output devices; and so forth. In anotherexample, in some implementations at least one of the following may beexcluded from server 300: memory units 210, communication modules 230,and power sources 240.

FIG. 4A is a block diagram illustrating a possible implementation ofcloud platform 400. In this example, cloud platform 400 may comprisecomputational node 500 a, computational node 500 b, computational node500 c and computational node 500 d. In some examples, a possibleimplementation of computational nodes 500 a, 500 b, 500 c and 500 d maycomprise server 300 as described in FIG. 3 . In some examples, apossible implementation of computational nodes 500 a, 500 b, 500 c and500 d may comprise computational node 500 as described in FIG. 5 .

FIG. 4B is a block diagram illustrating a possible implementation ofcloud platform 400. In this example, cloud platform 400 may comprise:one or more computational nodes 500, one or more shared memory modules410, one or more power sources 240, one or more node registrationmodules 420, one or more load balancing modules 430, one or moreinternal communication modules 440, and one or more externalcommunication modules 450. In some implementations, cloud platform 400may comprise additional components, while some components listed abovemay be excluded. For example, in some implementations cloud platform 400may also comprise at least one of the following: one or more user inputdevices; one or more output devices; and so forth. In another example,in some implementations at least one of the following may be excludedfrom cloud platform 400: shared memory modules 410, power sources 240,node registration modules 420, load balancing modules 430, internalcommunication modules 440, and external communication modules 450.

FIG. 5 is a block diagram illustrating a possible implementation ofcomputational node 500. In this example, computational node 500 maycomprise: one or more memory units 210, one or more processing units220, one or more shared memory access modules 510, one or more powersources 240, one or more internal communication modules 440, and one ormore external communication modules 450. In some implementations,computational node 500 may comprise additional components, while somecomponents listed above may be excluded. For example, in someimplementations computational node 500 may also comprise at least one ofthe following: one or more user input devices; one or more outputdevices; and so forth. In another example, in some implementations atleast one of the following may be excluded from computational node 500:memory units 210, shared memory access modules 510, power sources 240,internal communication modules 440, and external communication modules450.

In some embodiments, internal communication modules 440 and externalcommunication modules 450 may be implemented as a combined communicationmodule, such as communication modules 230. In some embodiments, onepossible implementation of cloud platform 400 may comprise server 300.In some embodiments, one possible implementation of computational node500 may comprise server 300. In some embodiments, one possibleimplementation of shared memory access modules 510 may comprise usinginternal communication modules 440 to send information to shared memorymodules 410 and/or receive information from shared memory modules 410.In some embodiments, node registration modules 420 and load balancingmodules 430 may be implemented as a combined module.

In some embodiments, the one or more shared memory modules 410 may beaccessed by more than one computational node. Therefore, shared memorymodules 410 may allow information sharing among two or morecomputational nodes 500. In some embodiments, the one or more sharedmemory access modules 510 may be configured to enable access ofcomputational nodes 500 and/or the one or more processing units 220 ofcomputational nodes 500 to shared memory modules 410. In some examples,computational nodes 500 and/or the one or more processing units 220 ofcomputational nodes 500, may access shared memory modules 410, forexample using shared memory access modules 510, in order to perform atleast one of: executing software programs stored on shared memorymodules 410, store information in shared memory modules 410, retrieveinformation from the shared memory modules 410.

In some embodiments, the one or more node registration modules 420 maybe configured to track the availability of the computational nodes 500.In some examples, node registration modules 420 may be implemented as: asoftware program, such as a software program executed by one or more ofthe computational nodes 500; a hardware solution; a combined softwareand hardware solution; and so forth. In some implementations, noderegistration modules 420 may communicate with computational nodes 500,for example using internal communication modules 440. In some examples,computational nodes 500 may notify node registration modules 420 oftheir status, for example by sending messages: at computational node 500startup; at computational node 500 shutdown; at constant intervals; atselected times; in response to queries received from node registrationmodules 420; and so forth. In some examples, node registration modules420 may query about computational nodes 500 status, for example bysending messages: at node registration module 420 startup; at constantintervals; at selected times; and so forth.

In some embodiments, the one or more load balancing modules 430 may beconfigured to divide the work load among computational nodes 500. Insome examples, load balancing modules 430 may be implemented as: asoftware program, such as a software program executed by one or more ofthe computational nodes 500; a hardware solution; a combined softwareand hardware solution; and so forth. In some implementations, loadbalancing modules 430 may interact with node registration modules 420 inorder to obtain information regarding the availability of thecomputational nodes 500. In some implementations, load balancing modules430 may communicate with computational nodes 500, for example usinginternal communication modules 440. In some examples, computationalnodes 500 may notify load balancing modules 430 of their status, forexample by sending messages: at computational node 500 startup; atcomputational node 500 shutdown; at constant intervals; at selectedtimes; in response to queries received from load balancing modules 430;and so forth. In some examples, load balancing modules 430 may queryabout computational nodes 500 status, for example by sending messages:at load balancing module 430 startup; at constant intervals; at selectedtimes; and so forth.

In some embodiments, the one or more internal communication modules 440may be configured to receive information from one or more components ofcloud platform 400, and/or to transmit information to one or morecomponents of cloud platform 400. For example, control signals and/orsynchronization signals may be sent and/or received through internalcommunication modules 440. In another example, input information forcomputer programs, output information of computer programs, and/orintermediate information of computer programs, may be sent and/orreceived through internal communication modules 440. In another example,information received though internal communication modules 440 may bestored in memory units 210, in shared memory units 410, and so forth. Inan additional example, information retrieved from memory units 210and/or shared memory units 410 may be transmitted using internalcommunication modules 440. In another example, input data may betransmitted and/or received using internal communication modules 440.Examples of such input data may include input data inputted by a userusing user input devices.

In some embodiments, the one or more external communication modules 450may be configured to receive and/or to transmit information. Forexample, control signals may be sent and/or received through externalcommunication modules 450. In another example, information receivedthough external communication modules 450 may be stored in memory units210, in shared memory units 410, and so forth. In an additional example,information retrieved from memory units 210 and/or shared memory units410 may be transmitted using external communication modules 450. Inanother example, input data may be transmitted and/or received usingexternal communication modules 450. Examples of such input data mayinclude: input data inputted by a user using user input devices;information captured from the environment of apparatus 200 using one ormore sensors; and so forth. Examples of such sensors may include: audiosensors 250; image sensors 260; motion sensors 270; positioning sensors275; chemical sensors; temperature sensors; barometers; and so forth.

In some embodiments, one or more instances of apparatus 200 may bemounted and/or configured to be mounted on a vehicle. The instances maybe mounted and/or configured to be mounted on one or more sides of thevehicle (such as front, back, left, right, and so forth). The instancesmay be configured to use image sensors 260 to capture and/or analyzeimages of the environment of the vehicle, of the exterior of thevehicle, of the interior of the vehicle, and so forth. Multiple suchvehicles may be equipped with such apparatuses, and information based onimages captured using the apparatuses may be gathered from the multiplevehicles. Additionally or alternatively, information from other sensorsmay be collected and/or analyzed, such as audio sensors 250, motionsensors 270, positioning sensors 275, and so forth. Additionally oralternatively, one or more additional instances of apparatus 200 may bepositioned and/or configured to be positioned in an environment of thevehicles (such as a street, a parking area, and so forth), and similarinformation from the additional instances may be gathered and/oranalyzed. The information captured and/or collected may be analyzed atthe vehicle and/or at the apparatuses in the environment of the vehicle,for example using apparatus 200. Additionally or alternatively, theinformation captured and/or collected may be transmitted to an externaldevice (such as server 300, cloud platform 400, etc.), possibly aftersome preprocessing, and the external device may gather and/or analyzethe information.

In some embodiments, a method, such as methods 700, 800, 900, 1000,1100, 1200, 1300, 1400, 1500, 1600, 1700, 1800, 1900, 2000, 2100, 2200,and 2300 may comprise of one or more steps. In some examples, a method,as well as all individual steps therein, may be performed by variousaspects of apparatus 200, server 300, cloud platform 400, computationalnode 500, and so forth. For example, the method may be performed byprocessing units 220 executing software instructions stored withinmemory units 210 and/or within shared memory modules 410. In someexamples, a method, as well as all individual steps therein, may beperformed by a dedicated hardware. In some examples, computer readablemedium (such as a non-transitory computer readable medium) may storedata and/or computer implementable instructions for carrying out amethod. Some non-limiting examples of possible execution manners of amethod may include continuous execution (for example, returning to thebeginning of the method once the method normal execution ends),periodically execution, executing the method at selected times,execution upon the detection of a trigger (some non-limiting examples ofsuch trigger may include a trigger from a user, a trigger from anothermethod, a trigger from an external device, etc.), and so forth.

In some embodiments, machine learning algorithms (also referred to asmachine learning models in the present disclosure) may be trained usingtraining examples, for example in the cases described below. Somenon-limiting examples of such machine learning algorithms may includeclassification algorithms, data regressions algorithms, imagesegmentation algorithms, visual detection algorithms (such as objectdetectors, face detectors, person detectors, motion detectors, edgedetectors, etc.), visual recognition algorithms (such as facerecognition, person recognition, object recognition, etc.), speechrecognition algorithms, mathematical embedding algorithms, naturallanguage processing algorithms, support vector machines, random forests,nearest neighbors algorithms, deep learning algorithms, artificialneural network algorithms, convolutional neural network algorithms,recurrent neural network algorithms, linear algorithms, non-linearalgorithms, ensemble algorithms, and so forth. For example, a trainedmachine learning algorithm may comprise an inference model, such as apredictive model, a classification model, a regression model, aclustering model, a segmentation model, an artificial neural network(such as a deep neural network, a convolutional neural network, arecurrent neural network, etc.), a random forest, a support vectormachine, and so forth. In some examples, the training examples mayinclude example inputs together with the desired outputs correspondingto the example inputs. Further, in some examples, training machinelearning algorithms using the training examples may generate a trainedmachine learning algorithm, and the trained machine learning algorithmmay be used to estimate outputs for inputs not included in the trainingexamples. In some examples, engineers, scientists, processes andmachines that train machine learning algorithms may further usevalidation examples and/or test examples. For example, validationexamples and/or test examples may include example inputs together withthe desired outputs corresponding to the example inputs, a trainedmachine learning algorithm and/or an intermediately trained machinelearning algorithm may be used to estimate outputs for the exampleinputs of the validation examples and/or test examples, the estimatedoutputs may be compared to the corresponding desired outputs, and thetrained machine learning algorithm and/or the intermediately trainedmachine learning algorithm may be evaluated based on a result of thecomparison. In some examples, a machine learning algorithm may haveparameters and hyper parameters, where the hyper parameters are setmanually by a person or automatically by an process external to themachine learning algorithm (such as a hyper parameter search algorithm),and the parameters of the machine learning algorithm are set by themachine learning algorithm according to the training examples. In someimplementations, the hyper-parameters are set according to the trainingexamples and the validation examples, and the parameters are setaccording to the training examples and the selected hyper-parameters.

In some embodiments, trained machine learning algorithms (also referredto as trained machine learning models in the present disclosure) may beused to analyze inputs and generate outputs, for example in the casesdescribed below. In some examples, a trained machine learning algorithmmay be used as an inference model that when provided with an inputgenerates an inferred output. For example, a trained machine learningalgorithm may include a classification algorithm, the input may includea sample, and the inferred output may include a classification of thesample (such as an inferred label, an inferred tag, and so forth). Inanother example, a trained machine learning algorithm may include aregression model, the input may include a sample, and the inferredoutput may include an inferred value for the sample. In yet anotherexample, a trained machine learning algorithm may include a clusteringmodel, the input may include a sample, and the inferred output mayinclude an assignment of the sample to at least one cluster. In anadditional example, a trained machine learning algorithm may include aclassification algorithm, the input may include an image, and theinferred output may include a classification of an item depicted in theimage. In yet another example, a trained machine learning algorithm mayinclude a regression model, the input may include an image, and theinferred output may include an inferred value for an item depicted inthe image (such as an estimated property of the item, such as size,volume, age of a person depicted in the image, cost of a productdepicted in the image, and so forth). In an additional example, atrained machine learning algorithm may include an image segmentationmodel, the input may include an image, and the inferred output mayinclude a segmentation of the image. In yet another example, a trainedmachine learning algorithm may include an object detector, the input mayinclude an image, and the inferred output may include one or moredetected objects in the image and/or one or more locations of objectswithin the image. In some examples, the trained machine learningalgorithm may include one or more formulas and/or one or more functionsand/or one or more rules and/or one or more procedures, the input may beused as input to the formulas and/or functions and/or rules and/orprocedures, and the inferred output may be based on the outputs of theformulas and/or functions and/or rules and/or procedures (for example,selecting one of the outputs of the formulas and/or functions and/orrules and/or procedures, using a statistical measure of the outputs ofthe formulas and/or functions and/or rules and/or procedures, and soforth).

In some embodiments, artificial neural networks may be configured toanalyze inputs and generate corresponding outputs. Some non-limitingexamples of such artificial neural networks may comprise shallowartificial neural networks, deep artificial neural networks, feedbackartificial neural networks, feed forward artificial neural networks,autoencoder artificial neural networks, probabilistic artificial neuralnetworks, time delay artificial neural networks, convolutionalartificial neural networks, recurrent artificial neural networks, longshort term memory artificial neural networks, and so forth. In someexamples, an artificial neural network may be configured manually. Forexample, a structure of the artificial neural network may be selectedmanually, a type of an artificial neuron of the artificial neuralnetwork may be selected manually, a parameter of the artificial neuralnetwork (such as a parameter of an artificial neuron of the artificialneural network) may be selected manually, and so forth. In someexamples, an artificial neural network may be configured using a machinelearning algorithm. For example, a user may select hyper-parameters forthe an artificial neural network and/or the machine learning algorithm,and the machine learning algorithm may use the hyper-parameters andtraining examples to determine the parameters of the artificial neuralnetwork, for example using back propagation, using gradient descent,using stochastic gradient descent, using mini-batch gradient descent,and so forth. In some examples, an artificial neural network may becreated from two or more other artificial neural networks by combiningthe two or more other artificial neural networks into a singleartificial neural network.

In some embodiments, analyzing audio data (for example, by the methods,steps and modules described herein) may comprise analyzing the audiodata to obtain a preprocessed audio data, and subsequently analyzing theaudio data and/or the preprocessed audio data to obtain the desiredoutcome. One of ordinary skill in the art will recognize that thefollowings are examples, and that the audio data may be preprocessedusing other kinds of preprocessing methods. In some examples, the audiodata may be preprocessed by transforming the audio data using atransformation function to obtain a transformed audio data, and thepreprocessed audio data may comprise the transformed audio data. Forexample, the transformation function may comprise a multiplication of avectored time series representation of the audio data with atransformation matrix. For example, the transformation function maycomprise convolutions, audio filters (such as low-pass filters,high-pass filters, band-pass filters, all-pass filters, etc.), nonlinearfunctions, and so forth. In some examples, the audio data may bepreprocessed by smoothing the audio data, for example using Gaussianconvolution, using a median filter, and so forth. In some examples, theaudio data may be preprocessed to obtain a different representation ofthe audio data. For example, the preprocessed audio data may comprise: arepresentation of at least part of the audio data in a frequency domain;a Discrete Fourier Transform of at least part of the audio data; aDiscrete Wavelet Transform of at least part of the audio data; atime/frequency representation of at least part of the audio data; aspectrogram of at least part of the audio data; a log spectrogram of atleast part of the audio data; a Mel-Frequency Cepstrum of at least partof the audio data; a sonogram of at least part of the audio data; aperiodogram of at least part of the audio data; a representation of atleast part of the audio data in a lower dimension; a lossyrepresentation of at least part of the audio data; a losslessrepresentation of at least part of the audio data; a time order seriesof any of the above; any combination of the above; and so forth. In someexamples, the audio data may be preprocessed to extract audio featuresfrom the audio data. Some non-limiting examples of such audio featuresmay include: auto-correlation; number of zero crossings of the audiosignal; number of zero crossings of the audio signal centroid; MP3 basedfeatures; rhythm patterns; rhythm histograms; spectral features, such asspectral centroid, spectral spread, spectral skewness, spectralkurtosis, spectral slope, spectral decrease, spectral roll-off, spectralvariation, etc.; harmonic features, such as fundamental frequency,noisiness, inharmonicity, harmonic spectral deviation, harmonic spectralvariation, tristimulus, etc.; statistical spectrum descriptors; waveletfeatures; higher level features; perceptual features, such as totalloudness, specific loudness, relative specific loudness, sharpness,spread, etc.; energy features, such as total energy, harmonic partenergy, noise part energy, etc.; temporal features; and so forth.

In some embodiments, analyzing audio data (for example, by the methods,steps and modules described herein) may comprise analyzing the audiodata and/or the preprocessed audio data using one or more rules,functions, procedures, artificial neural networks, speech recognitionalgorithms, speaker recognition algorithms, speaker diarizationalgorithms, audio segmentation algorithms, noise cancelling algorithms,source separation algorithms, inference models, and so forth. Somenon-limiting examples of such inference models may include: an inferencemodel preprogrammed manually; a classification model; a regressionmodel; a result of training algorithms, such as machine learningalgorithms and/or deep learning algorithms, on training examples, wherethe training examples may include examples of data instances, and insome cases, a data instance may be labeled with a corresponding desiredlabel and/or result; and so forth.

In some embodiments, analyzing one or more images (for example, by themethods, steps and modules described herein) may comprise analyzing theone or more images to obtain a preprocessed image data, and subsequentlyanalyzing the one or more images and/or the preprocessed image data toobtain the desired outcome. One of ordinary skill in the art willrecognize that the followings are examples, and that the one or moreimages may be preprocessed using other kinds of preprocessing methods.In some examples, the one or more images may be preprocessed bytransforming the one or more images using a transformation function toobtain a transformed image data, and the preprocessed image data maycomprise the transformed image data. For example, the transformed imagedata may comprise one or more convolutions of the one or more images.For example, the transformation function may comprise one or more imagefilters, such as low-pass filters, high-pass filters, band-pass filters,all-pass filters, and so forth. In some examples, the transformationfunction may comprise a nonlinear function. In some examples, the one ormore images may be preprocessed by smoothing at least parts of the oneor more images, for example using Gaussian convolution, using a medianfilter, and so forth. In some examples, the one or more images may bepreprocessed to obtain a different representation of the one or moreimages. For example, the preprocessed image data may comprise: arepresentation of at least part of the one or more images in a frequencydomain; a Discrete Fourier Transform of at least part of the one or moreimages; a Discrete Wavelet Transform of at least part of the one or moreimages; a time/frequency representation of at least part of the one ormore images; a representation of at least part of the one or more imagesin a lower dimension; a lossy representation of at least part of the oneor more images; a lossless representation of at least part of the one ormore images; a time ordered series of any of the above; any combinationof the above; and so forth. In some examples, the one or more images maybe preprocessed to extract edges, and the preprocessed image data maycomprise information based on and/or related to the extracted edges. Insome examples, the one or more images may be preprocessed to extractimage features from the one or more images. Some non-limiting examplesof such image features may comprise information based on and/or relatedto: edges; corners; blobs; ridges; Scale Invariant Feature Transform(SIFT) features; temporal features; and so forth.

In some embodiments, analyzing one or more images (for example, by themethods, steps and modules described herein) may comprise analyzing theone or more images and/or the preprocessed image data using one or morerules, functions, procedures, artificial neural networks, objectdetection algorithms, face detection algorithms, visual event detectionalgorithms, action detection algorithms, motion detection algorithms,background subtraction algorithms, inference models, and so forth. Somenon-limiting examples of such inference models may include: an inferencemodel preprogrammed manually; a classification model; a regressionmodel; a result of training algorithms, such as machine learningalgorithms and/or deep learning algorithms, on training examples, wherethe training examples may include examples of data instances, and insome cases, a data instance may be labeled with a corresponding desiredlabel and/or result; and so forth.

In some embodiments, analyzing one or more images (for example, by themethods, steps and modules described herein) may comprise analyzingpixels, voxels, point cloud, range data, etc. included in the one ormore images.

In some embodiments, one or more images captured using one or more imagesensors from an environment of a first vehicle may be obtained, forexample using step 710 as described below. Further, in some examples,the one or more images may be analyzed to detect a second vehicle, forexample using step 720 as described below. Further, in some examples,the one or more images may be analyzed to determine that the secondvehicle is connected to an item. Further, in some examples, the one ormore images may be analyzed to determine a state of the item. Further,in some examples, in response to a first determined state of the item,the first vehicle may be caused to initiate an action responding to thesecond vehicle, and in response to a second determined state of theitem, causing the first vehicle to initiate the action may be withheldand/or forgone.

In some embodiments, the one or more images obtained by step 710 may beanalyzed to determine that the second vehicle detected by step 720 isconnected to an item of a particular type. For example, step 730 mayanalyze the one or more images obtained by step 710 to determine thatthe second vehicle detected by step 720 is connected to a crane, step830 may analyze the one or more images obtained by step 710 to determinethat the second vehicle detected by step 720 is connected to a lift,step 930 may analyze the one or more images obtained by step 710 todetermine that the second vehicle detected by step 720 is connected toan outrigger, step 1030 may analyze the one or more images obtained bystep 710 to determine that the second vehicle detected by step 720 isconnected to a pump, step 1130 may analyze the one or more imagesobtained by step 710 to determine that the second vehicle detected bystep 720 is connected to a pipe, and so forth. For example, a machinelearning model may be trained using training examples to determinewhether vehicles are connected to items of particular types from imagesand/or videos, and the trained machine learning model may be used toanalyze the one or more images obtained by step 710 and determine thatthe second vehicle detected by step 720 is connected to an item of aparticular type. An example of such training example may include animage and/or a video depicting a vehicle, together with a labelindicating whether the depicted vehicle is connected to an item of aparticular type. In another example, an artificial neural network (suchas a deep neural networks, convolutional neural networks, etc.) may beconfigured to determine whether vehicles are connected to items ofparticular types from images and/or videos, and the artificial neuralnetwork may be used to analyze the one or more images obtained by step710 and determine that the second vehicle detected by step 720 isconnected to an item of a particular type. In yet another example, imageclassification algorithms may be used to analyze the one or more imagesobtained by step 710 and determine that the second vehicle detected bystep 720 is connected to an item of a particular type.

In some embodiments, the one or more images obtained by step 710 may beanalyzed to determine a state of an item. For example, step 740 mayanalyze the one or more images obtained by step 710 to determine a stateof the crane of step 730, step 840 may analyze the one or more imagesobtained by step 710 to determine a state of the lift of step 830, step940 may analyze the one or more images obtained by step 710 to determinea state of the outrigger of step 930, step 1040 may analyze the one ormore images obtained by step 710 to determine a state of the pump ofstep 1030, step 1140 may analyze the one or more images obtained by step710 to determine a state of the pipe of step 1130, step 1330 may analyzethe one or more images obtained by step 710 to determine a state of adoor of the second vehicle detected by step 720, step 1430 may analyzethe one or more images obtained by step 710 to determine a state of auser associated with the second vehicle detected by step 720, step 1730may analyze the one or more images obtained by step 710 to determine astate of a hood of the second vehicle detected by step 720, step 1830may analyze the one or more images obtained by step 710 to determine astate of a trunk lid of the second vehicle detected by step 720, and soforth. For example, a machine learning model may be trained usingtraining examples to determine states of items from images and/orvideos, and the trained machine learning model may be used to analyzethe one or more images obtained by step 710 and determine the state ofthe item. An example of such training example may include an imageand/or a video depicting an item connected to a vehicle, together with alabel indicating the state of the depicted item. In another example, anartificial neural network (such as a deep neural networks, convolutionalneural networks, etc.) may be configured to determine state of an itemfrom images and/or videos, and the artificial neural network may be usedto analyze the one or more images obtained by step 710 and determine thestate of the item. In yet another example, image classificationalgorithms may be used to analyze the one or more images obtained bystep 710 and determine the state of the item. Some other non-limitingexamples of steps for determining a state of an item are describedbelow.

In some embodiments, for example in response to a first determined stateof the item (such as a crane, a lift, an outrigger, a pump, a pipe, adoor of a vehicle, a user associated with a vehicle, a hood of avehicle, a trunk lid of a vehicle, etc.), the first vehicle may becaused to initiate an action responding to the item and/or to the secondvehicle (and in response to a second determined state of the item,causing the first vehicle to initiate the action may be withheld and/orforgone). Some non-limiting examples of such action may includesignaling, stopping, changing a speed of the first vehicle, changing amotion direction of the first vehicle, passing the item, passing thesecond vehicle, forgoing passing the second vehicle, keeping a minimaldistance of at least a selected length from the item (for example, theselected length may be less than 100 feet, may be at least 100 feet, maybe at least 200 feet, etc.), keeping a minimal distance of at least aselected length from the second vehicle (for example, the selectedlength may be less than 100 feet, may be at least 100 feet, may be atleast 200 feet, etc.), turning, performing a U-turn, driving in reverse,generating an audible warning, and so forth. For example, a signal maybe transmitted to an external device (such as a device controlling thefirst vehicle, a device navigating the first vehicle, the first vehicle,a system within the first vehicle, etc.), and the signal may beconfigured to cause the external device to cause the first vehicle toinitiate the action responding to the second vehicle. In anotherexample, information related to the second vehicle may be provided tosuch external device, and the information may be configured to cause theexternal device to cause the first vehicle to initiate the actionresponding to the second vehicle. In some examples, the action may beselected based on the determined state of the item. For example, inresponse to a first state of the item, a first action may be selected,and in response to a second state of the item, a second action may beselected (where the second action may differ from the first action, andthe second state of the item may differ from the first state of theitem). In some examples, in response to a first state of the item, afirst action may be selected, and in response to a second state of theitem, the first action may be withheld and/or forgone (for example,causing the first vehicle to initiate the first action may be withheldand/or forgone).

FIG. 6 is a schematic illustration of example images of trucksconsistent with an embodiment of the present disclosure. In thisexample, truck 600A is connected to crane with boom 601A, outriggers602A and bucket 603A. Further, truck 600B is connected to crane withboom 601B, outriggers 602B, bucket 603B and person 604B in bucket 603B.While this example illustrates the cranes connected to buckets, andillustrates person 604B in bucket 603B, in some examples the cranes maybe connected to other devices, such as lifting hooks, electro magnets,pipes, and so forth, and in some examples the cranes may lift othertypes of (non-human) loads. In this example, analysis of the images ofthe trucks (for example, by methods 700 and/or 900 described below) mayindicate that the crane of truck 600A is not in use, and that the craneof truck 600B is in use. For example, the analysis of the images maydetermine that boom 601A is shorter than boom 601B that was extended inthe operation of the crane, may determine that boom 601A is in oneposition and/or orientation with respect to truck 600A and/or the groundwhile boom 601B is in another position and/or orientation with respectto truck 600B and/or the ground, as boom 601B may have been reorientedand/or repositioned in the operation of the crane, may determine thatoutriggers 602A are not in use while outriggers 602B are in use (forexample, by determining that outriggers 602A is in one position and/ororientation with respect to truck 600A and/or the ground whileoutriggers 602B is in another position and/or orientation with respectto truck 600B and/or the ground, by determining that outriggers 602A donot touch the ground while outriggers 602B touch the ground, etc.), maydetermine that bucket 603A is empty while bucket 603B carries person604B (or another device connected to the crane, such as a lifting hook,carries another kind of load), and so forth. Further, in some examples,based on the one or more of the above determinations, it may be furtherdetermined that the crane of truck 600A is not in use while the crane oftruck 600B is in use.

In some embodiments, systems, methods and computer readable media forcontrolling vehicles in response to cranes are provided. One challengeof autonomous driving is to determine when a standing vehicle hasstopped for a short period of time and therefore the autonomous vehicleneeds to wait for the standing vehicle to resume moving, and when thestanding vehicle has stopped for a longer period of time and thereforethe autonomous vehicle needs to pass the standing vehicle. In somecases, one indication of whether a vehicle with a crane has stopped fora longer period of time may include the state of the crane. For example,identifying that the crane is in use may indicate that the vehicle withthe crane has stopped for a longer period of time. The provided systems,methods and computer readable media for controlling vehicles may detecta vehicle connected to a crane, identify a state of the crane, andcontrol a response of an autonomous vehicle to the detected vehiclebased on the identified state of the crane.

FIG. 7 illustrates an example of a method 700 for controlling vehiclesin response to cranes. In this example, method 700 may comprise:obtaining images captured from an environment of a first vehicle (Step710); analyzing the images to detect a second vehicle (Step 720);analyzing the images to determine that the second vehicle is connectedto a crane (Step 730); analyzing the images to determine a state of thecrane (Step 740); in response to a first determined state of the crane,causing the first vehicle to initiate an action responding to the secondvehicle (Step 750), and in response to a second determined state of thecrane, forgoing causing the first vehicle to initiate the action. Insome implementations, method 700 may comprise one or more additionalsteps, while some of the steps listed above may be modified or excluded.In some implementations, one or more steps illustrated in FIG. 7 may beexecuted in a different order and/or one or more groups of steps may beexecuted simultaneously and/or a plurality of steps may be combined intosingle step and/or a single step may be broken down to a plurality ofsteps.

In some embodiments, one or more images captured using one or more imagesensors from an environment of a first vehicle may be obtained. Further,in some examples, the one or more images may be analyzed to detect asecond vehicle. Further, in some examples, the one or more images may beanalyzed to determine that the second vehicle is connected to a crane.Further, in some examples, the one or more images may be analyzed todetermine a state of the crane. Further, in some examples, in responseto a first determined state of the crane, the first vehicle may becaused to initiate an action responding to the second vehicle.

In some embodiments, step 710 may comprise obtaining one or more imagescaptured using one or more image sensors from an environment of a firstvehicle. For example, at least part of the one or more image sensors maybe part of the first vehicle, may be attached to the first vehicle, maybe part of or attached to another vehicle in the environment of thefirst vehicle, may be stationary cameras in the environment of the firstvehicle (such as street cameras, security cameras, traffic monitoringcameras, etc.), may be attached to a drone in the environment of thefirst vehicle, and so forth. For example, step 710 may read at leastpart of the one or more images may be from memory (such as memory unit210, shared memory module 410, remote storage 140, NAS 150, etc.), maybe received through a communication network using a communication device(such as communication module 230, internal communication module 440,external communication module 450, etc.), may be received from anexternal device, may be captured using the one or more image sensors(such as image sensors 260), and so forth.

In some embodiments, audio data captured using one or more audio sensorsfrom an environment of the first vehicle may be obtained. For example,at least part of the one or more audio sensors may be part of the firstvehicle, may be attached to the first vehicle, may be part of orattached to another vehicle in the environment of the first vehicle, maybe stationary audio sensors in the environment of the first vehicle(such as street microphones), may be attached to a drone in theenvironment of the first vehicle, and so forth. For example, at leastpart of the audio data may be read from memory (such as memory unit 210,shared memory module 410, remote storage 140, NAS 150, etc.), may bereceived through a communication network using a communication device(such as memory unit 210, shared memory module 410, remote storage 140,NAS 150, etc.), may be received from an external device, may be capturedusing the one or more audio sensors (such as audio sensors 250), and soforth.

In some embodiments, step 720 may comprise analyzing the one or moreimages obtained by step 710 to detect a second vehicle. Somenon-limiting examples of such second vehicle may include a bus, a taxi,a passenger car, a truck, a garbage truck, a motorcycle, a bicycle, atwo-wheeler vehicle, a motor vehicle, and so forth. For example, amachine learning model may be trained using training examples to detectvehicles in images and/or videos, and step 720 may use the trainedmachine learning model to analyze the one or more images obtained bystep 710 and detect the second vehicle in the one or more images. Anexample of such training example may include an image and/or a video,together with a label indicating whether a vehicle appears in the imageand/or in the video, and/or together with a label indicating a positionof the vehicle in the image and/or in the video. In another example, anartificial neural network (such as a deep neural networks, convolutionalneural networks, etc.) may be configured to detect vehicles in imagesand/or videos, and step 720 may use the artificial neural network toanalyze the one or more images obtained by step 710 and detect thesecond vehicle in the one or more images. In yet another example, step720 may use object detection algorithms to analyze the one or moreimages obtained by step 710 and detect the second vehicle in the one ormore images.

In some embodiments, step 730 may comprise analyzing the one or moreimages obtained by step 710 to determine that the second vehicledetected by step 720 is connected to a crane. For example, a machinelearning model may be trained using training examples to determinewhether vehicles are connected to cranes from images and/or videos, andstep 730 may use the trained machine learning model to analyze the oneor more images obtained by step 710 and determine that the secondvehicle detected by step 720 is connected to a crane. An example of suchtraining example may include an image and/or a video depicting avehicle, together with a label indicating whether the depicted vehicleis connected to a crane. In another example, an artificial neuralnetwork (such as a deep neural networks, convolutional neural networks,etc.) may be configured to determine whether vehicles are connected tocranes from images and/or videos, and step 730 may use the artificialneural network to analyze the one or more images obtained by step 710and determine that the second vehicle detected by step 720 is connectedto a crane. In yet another example, step 730 may use imageclassification algorithms to analyze the one or more images obtained bystep 710 and determine that the second vehicle detected by step 720 isconnected to a crane.

In some embodiments, step 740 may comprise analyzing the one or moreimages obtained by step 710 to determine a state of the crane of step730. Some non-limiting examples of such state of the crane may includecrane in use, crane not in use, crane holding object, crane not holdingobject, crane tied, and so forth. For example, a machine learning modelmay be trained using training examples to determine states of cranesfrom images and/or videos, and step 740 may use the trained machinelearning model to analyze the one or more images obtained by step 710and determine the state of the crane of step 730. An example of suchtraining example may include an image and/or a video depicting a craneconnected to a vehicle, together with a label indicating the state ofthe depicted crane. In another example, an artificial neural network(such as a deep neural networks, convolutional neural networks, etc.)may be configured to determine states of cranes from images and/orvideos, and step 740 may use the artificial neural network to analyzethe one or more images obtained by step 710 and determine the state ofthe crane of step 730. In yet another example, step 740 may use imageclassification algorithms to analyze the one or more images obtained bystep 710 and determine the state of the crane of step 730. Some othernon-limiting examples of steps that may be used by step 740 fordetermining the state of the crane of step 730 are described below.

In some examples, an orientation of at least part of the crane (inrelation to at least part of the second vehicle, in relation to theground, in relation to the horizon, in relation to an object in theenvironment, in relation to the at least part of the first vehicle, inrelation to the image sensor, etc.) may be determined, and thedetermined state of the crane may be based on the determined orientationof the at least part of the crane. For example, a machine learning modelmay be trained using training examples to determine orientation of partsof cranes by analyzing images and/or videos, and the trained machinelearning model may be used to determine the orientation of the at leastpart of the crane from the one or more images. In another example, anartificial neural network (such as a deep neural networks, convolutionalneural networks, etc.) may be configured to determine orientation ofparts of cranes by analyzing images and/or videos, and the configuredartificial neural network may be used to determine the orientation ofthe at least part of the crane from the one or more images. In yetanother example, information about the orientation of the at least partof the crane may be received from the second vehicle (for example, usinga point to point communication protocol, through a communication networkusing a communication device, through a centralized server, and soforth).

In some examples, a distance of at least part of the crane (from atleast part of the second vehicle, from the ground, from an object in theenvironment, from at least part of the first vehicle, from the imagesensor, etc.) may be determined, and the determined state of the cranemay be based on the determined distance of the at least part of thecrane. For example, a machine learning model may be trained usingtraining examples to determine distance of parts of cranes by analyzingimages and/or videos, and the trained machine learning model may be usedto determine the distance of the at least part of the crane from the oneor more images. In another example, an artificial neural network (suchas a deep neural networks, convolutional neural networks, etc.) may beconfigured to determine distance of parts of cranes by analyzing imagesand/or videos, and the configured artificial neural network may be usedto determine the distance of the at least part of the crane from the oneor more images. In yet another example, information about the distanceof the at least part of the crane may be received from the secondvehicle (for example, using a point to point communication protocol,through a communication network using a communication device, through acentralized server, and so forth).

In some examples, a motion of at least part of the crane (in relation toat least part of the second vehicle, in relation to the ground, inrelation to the horizon, in relation to an object in the environment, inrelation to the at least part of the first vehicle, in relation to theimage sensor, etc.) may be determined, and the determined state of thecrane may be based on the determined motion of the at least part of thecrane. For example, a machine learning model may be trained usingtraining examples to determine motion of parts of cranes by analyzingimages and/or videos, and the trained machine learning model may be usedto determine the motion of the at least part of the crane from the oneor more images. In another example, an artificial neural network (suchas a deep neural networks, convolutional neural networks, etc.) may beconfigured to determine motion of parts of cranes by analyzing imagesand/or videos, and the configured artificial neural network may be usedto determine the motion of the at least part of the crane from the oneor more images. In yet another example, information about the motion ofthe at least part of the crane may be received from the second vehicle(for example, using a point to point communication protocol, through acommunication network using a communication device, through acentralized server, and so forth).

In some examples, a length of at least part of the crane may bedetermined, and the determined state of the crane may be based on thedetermined length of the at least part of the crane. For example, amachine learning model may be trained using training examples todetermine lengths of parts of cranes by analyzing images and/or videos,and the trained machine learning model may be used to determine thelength of the at least part of the crane from the one or more images. Inanother example, an artificial neural network (such as a deep neuralnetworks, convolutional neural networks, etc.) may be configured todetermine lengths of parts of cranes by analyzing images and/or videos,and the configured artificial neural network may be used to determinethe length of the at least part of the crane from the one or moreimages. In yet another example, information about the length of the atleast part of the crane may be received from the second vehicle (forexample, using a point to point communication protocol, through acommunication network using a communication device, through acentralized server, and so forth).

In some embodiments, step 750 may comprise, for example in response to afirst state of the crane determined by step 740, causing the firstvehicle to initiate an action responding to the second vehicle detectedby step 720. Some non-limiting examples of such action may includesignaling, stopping, changing a speed of the first vehicle, changing amotion direction of the first vehicle, passing the second vehicle,forgoing passing the second vehicle, passing the crane, keeping aminimal distance of at least a selected length from the second vehicle(for example, the selected length may be less than 100 feet, may be atleast 100 feet, may be at least 200 feet, etc.), keeping a minimaldistance of at least a selected length from the crane (for example, theselected length may be less than 100 feet, may be at least 100 feet, maybe at least 200 feet, etc.), turning, performing a U-turn, driving inreverse, generating an audible warning, and so forth. For example, step750 may transmit a signal to an external device (such as a devicecontrolling the first vehicle, a device navigating the first vehicle,the first vehicle, a system within the first vehicle, etc.), and thesignal may be configured to cause the external device to cause the firstvehicle to initiate the action responding to the second vehicle detectedby step 720. In another example, step 750 may provide informationrelated to the second vehicle detected by step 720 (such as position,motion, acceleration, type, dimensions, etc.) to such external device,and the information may be configured to cause the external device tocause the first vehicle to initiate the action responding to the secondvehicle detected by step 720. In one example, in response to a secondstate of the crane determined by step 740, step 750 may cause the firstvehicle to initiate a second action, the second action may differ fromthe action.

In some examples, step 750 may select the action based on the state ofthe crane determined by step 740. For example, in response to a firststate of the crane determined by step 740, step 750 may select a firstaction, and in response to a second state of the crane determined bystep 740, step 750 may select a second action (where the second actionmay differ from the first action, and the second state of the crane maydiffer from the first state of the crane). In some examples, in responseto a first state of the crane determined by step 740, step 750 mayselect a first action, and in response to a second state of the cranedetermined by step 740, step 750 may withhold and/or forgo the firstaction (for example, step 750 may withhold and/or forgo causing thefirst vehicle to initiate the first action).

In some examples, a motion of the second vehicle may be determined. Insome examples, the motion of the second vehicle may be received from thesecond vehicle (for example, using a point to point communicationprotocol, through a communication network using a communication device,through a centralized server, and so forth). In some examples, radarimages and/or LiDAR images and/or depth images of the second vehicle maybe analyzed to determine the motion of the second vehicle. In someexamples, the one or more images may be analyzed to determine the motionof the second vehicle. For example a machine learning model may betrained using training examples to determine motion of objects byanalyzing images and/or videos, and the trained machine learning modelmay be used to determine the motion of the second vehicle from the oneor more images. In another example, an artificial neural network (suchas a deep neural networks, convolutional neural networks, etc.) may beconfigured to determine motion of objects by analyzing images and/orvideos, and the artificial neural network may be used to determine themotion of the second vehicle from the one or more images.

In some examples, in response to the determined motion of the secondvehicle and the first determined state of the crane, the first vehiclemay be caused to initiate the action (for example as described above).For example, in response to a first determined motion of the secondvehicle and a first determined state of the crane, the first vehicle maybe caused to initiate a first action; in response to a second determinedmotion of the second vehicle and the first determined state of thecrane, the first vehicle may be caused to initiate a second action; andin response to the first determined motion of the second vehicle and asecond determined state of the crane, the first vehicle may be caused toinitiate a third action. In another example, in response to a firstdetermined motion of the second vehicle and a first determined state ofthe crane, the first vehicle may be caused to initiate a first action;in response to a second determined motion of the second vehicle and thefirst determined state of the crane and/or in response to the firstdetermined motion of the second vehicle and a second determined state ofthe crane, causing the first vehicle to initiate the first action may bewithheld and/or forwent.

In some examples, it may be determining that the second vehicle issignaling. Further, in some examples, a type of the signaling of thesecond vehicle may be determined. Some non-limiting examples of suchtype of signaling may include signaling using light, signaling usingsound, signaling left, signaling right, signaling break, signalinghazard warning, signaling reversing, signaling warning, and so forth.For example, audio data captured using one or more audio sensors from anenvironment of the first vehicle may be analyzed (for example usingsignal processing algorithms, using a trained machine learning model,using an artificial neural network, etc.) to determine that the secondvehicle is signaling using it's horn and/or to determine a pattern or atype of honking. In another example, the one or more images may beanalyzed (for example, using a trained machine learning model, using anartificial neural network, etc.) to determine that the second vehicle issignaling and/or to determine a type of signaling. In yet anotherexample, indication that the second vehicle is signaling and/orinformation about the type of signaling may be received from the secondvehicle (for example, using a point to point communication protocol,through a communication network using a communication device, through acentralized server, and so forth).

In some examples, in response to the determination that the secondvehicle is signaling and/or the determined type of the signaling and/orthe first determined state of the crane, the first vehicle may be causedto initiate the action (for example as described above). For example, inresponse to a first determined type of the signaling and a firstdetermined state of the crane, the first vehicle may be caused toinitiate a first action; in response to a second determined type of thesignaling and the first determined state of the crane, the first vehiclemay be caused to initiate a second action; and in response to the firstdetermined type of the signaling and a second determined state of thecrane, the first vehicle may be caused to initiate a third action. Inanother example, in response to a first determined type of the signalingand a first determined state of the crane, the first vehicle may becaused to initiate a first action; in response to a second determinedtype of the signaling and the first determined state of the crane and/orin response to the first determined type of the signaling and a seconddetermined state of the crane, causing the first vehicle to initiate thefirst action may be withheld and/or forwent. In an additional example,in response to the determination that the second vehicle is signalingand a first determined state of the crane, the first vehicle may becaused to initiate a first action; and in response to the determinationthat the second vehicle is not signaling and the first determined stateof the crane, the first vehicle may be caused to initiate a secondaction. In yet another example, in response to the determination thatthe second vehicle is signaling and a first determined state of thecrane, the first vehicle may be caused to initiate a first action; andin response to the determination that the second vehicle is notsignaling and the first determined state of the crane, causing the firstvehicle to initiate the first action may be withheld and/or forwent.

In some examples, a position of the second vehicle may be determined.For example, the determined position may be in relation to the ground,in relation to a map, in relation to a road, in relation to a lane, inrelation to an object in the environment, in relation to the at leastpart of the first vehicle, in relation to the image sensor, and soforth. Further, in some examples, it may be determining that the secondvehicle is in a lane of the first vehicle. Further, in some examples, itmay be determining that the second vehicle is in a planned path of thefirst vehicle. For example, the one or more images may be analyzed (forexample using signal processing algorithms, using a trained machinelearning model, using an artificial neural network, etc.) to determine aposition of the second vehicle and/or to determine whether the secondvehicle is in a lane of the first vehicle and/or to determine whetherthe second vehicle is in a planned path of the first vehicle. In yetanother example, information related to the position and/or to the laneof the second vehicle is signaling and/or information about the type ofsignaling may be received from the second vehicle (for example, using apoint to point communication protocol, through a communication networkusing a communication device, through a centralized server, and soforth).

In some examples, in response to the determined position of the secondvehicle and the first determined state of the crane, the first vehiclemay be caused to initiate the action (for example as described above).For example, in response to a first determined position of the secondvehicle and a first determined state of the crane, the first vehicle maybe caused to initiate a first action; in response to a second determinedposition of the second vehicle and the first determined state of thecrane, the first vehicle may be caused to initiate a second action; andin response to the first determined position of the second vehicle and asecond determined state of the crane, the first vehicle may be caused toinitiate a third action. In another example, in response to a firstdetermined position of the second vehicle and a first determined stateof the crane, the first vehicle may be caused to initiate a firstaction; in response to a second determined position of the secondvehicle and the first determined state of the crane and/or in responseto the first determined position of the second vehicle and a seconddetermined state of the crane, causing the first vehicle to initiate thefirst action may be withheld and/or forwent. In an additional example,in response to the determination that the second vehicle is in a laneand/or a planned path of the first vehicle and a first determined stateof the crane, the first vehicle may be caused to initiate a firstaction; and in response to the determination that the second vehicle isnot in a lane and/or is not in a planned path of the first vehicle andthe first determined state of the crane, the first vehicle may be causedto initiate a second action. In yet another example, in response to thedetermination that the second vehicle is in a lane and/or a planned pathof the first vehicle and a first determined state of the crane, thefirst vehicle may be caused to initiate a first action; and in responseto the determination that the second vehicle is not in a lane and/or isnot in a planned path of the first vehicle and the first determinedstate of the crane, causing the first vehicle to initiate the firstaction may be withheld and/or forwent.

In some embodiments, systems, methods and computer readable media forcontrolling vehicles in response to lifts are provided. One challenge ofautonomous driving is to determine when a standing vehicle has stoppedfor a short period of time and therefore the autonomous vehicle needs towait for the standing vehicle to resume moving, and when the standingvehicle has stopped for a longer period of time and therefore theautonomous vehicle needs to pass the standing vehicle. In some cases,one indication of whether a vehicle with a lift has stopped for a longerperiod of time may include the state of the lift. For example,identifying that the lift is in use may indicate that the vehicle withthe lift has stopped for a longer period of time. The provided systems,methods and computer readable media for controlling vehicles may detecta vehicle connected to a lift, identify a state of the lift, and controla response of an autonomous vehicle to the detected vehicle based on theidentified state of the lift.

FIG. 8 illustrates an example of a method 800 for controlling vehiclesin response to lifts. In this example, method 800 may comprise:obtaining images captured from an environment of a first vehicle (Step710); analyzing the images to detect a second vehicle (Step 720);analyzing the images to determine that the second vehicle is connectedto a lift (Step 830); analyzing the images to determine a state of thelift (Step 840); in response to a first determined state of the lift,causing the first vehicle to initiate an action responding to the secondvehicle (Step 850), and in response to a second determined state of thelift, forgoing causing the first vehicle to initiate the action. In someimplementations, method 800 may comprise one or more additional steps,while some of the steps listed above may be modified or excluded. Insome implementations, one or more steps illustrated in FIG. 8 may beexecuted in a different order and/or one or more groups of steps may beexecuted simultaneously and/or a plurality of steps may be combined intosingle step and/or a single step may be broken down to a plurality ofsteps.

In some embodiments, one or more images captured using one or more imagesensors from an environment of a first vehicle may be obtained, forexample as described above. Further, in some examples, the one or moreimages may be analyzed to detect a second vehicle, for example asdescribed above. Further, in some examples, the one or more images maybe analyzed to determine that the second vehicle is connected to a lift.Further, in some examples, the one or more images may be analyzed todetermine a state of the lift. Further, in some examples, in response toa first determined state of the lift, the first vehicle may be caused toinitiate an action responding to the second vehicle.

In some embodiments, step 830 may analyze the one or more imagesobtained by step 710 to determine that the second vehicle detected bystep 720 is connected to a lift. For example, a machine learning modelmay be trained using training examples to determine whether vehicles areconnected to lifts from images and/or videos, and step 830 may use thetrained machine learning model to analyze the one or more imagesobtained by step 710 and determine that the second vehicle detected bystep 720 is connected to a lift. An example of such training example mayinclude an image and/or a video depicting a vehicle, together with alabel indicating whether the depicted vehicle is connected to a lift. Inanother example, an artificial neural network (such as a deep neuralnetworks, convolutional neural networks, etc.) may be configured todetermine whether vehicles are connected to lifts from images and/orvideos, and step 830 may use the artificial neural network to analyzethe one or more images obtained by step 710 and determine that thesecond vehicle detected by step 720 is connected to a lift. In yetanother example, step 830 may use image classification algorithms toanalyze the one or more images obtained by step 710 and determine thatthe second vehicle detected by step 720 is connected to a lift.

In some embodiments, step 840 may comprise analyzing the one or moreimages obtained by step 710 to determine a state of the lift of step830. Some non-limiting examples of such state of the lift may includelift in use, lift not in use, lift holding object, lift not holdingobject, and so forth. For example, a machine learning model may betrained using training examples to determine states of lifts from imagesand/or videos, and step 840 may use the trained machine learning modelto analyze the one or more images obtained by step 710 and determine thestate of the lift of step 830. An example of such training example mayinclude an image and/or a video depicting a lift connected to a vehicle,together with a label indicating the state of the depicted lift. Inanother example, an artificial neural network (such as a deep neuralnetworks, convolutional neural networks, etc.) may be configured todetermine states of lifts from images and/or videos, and step 840 maythe artificial neural network to analyze the one or more images obtainedby step 710 and determine the state of the lift of step 830. In yetanother example, step 840 may use image classification algorithms toanalyze the one or more images obtained by step 710 and determine thestate of the lift of step 830. Some other non-limiting examples of stepsthat may be used by step 840 for determining the state of the lift ofstep 830 are described below.

In some examples, an orientation of at least part of the lift (inrelation to at least part of the second vehicle, in relation to theground, in relation to the horizon, in relation to an object in theenvironment, in relation to the at least part of the first vehicle, inrelation to the image sensor, etc.) may be determined, and thedetermined state of the lift may be based on the determined orientationof the at least part of the lift. For example, a machine learning modelmay be trained using training examples to determine orientation of partsof lifts by analyzing images and/or videos, and the trained machinelearning model may be used to determine the orientation of the at leastpart of the lift from the one or more images. In another example, anartificial neural network (such as a deep neural networks, convolutionalneural networks, etc.) may be configured to determine orientation ofparts of lifts by analyzing images and/or videos, and the configuredartificial neural network may be used to determine the orientation ofthe at least part of the lift from the one or more images. In yetanother example, information about the orientation of the at least partof the lift may be received from the second vehicle (for example, usinga point to point communication protocol, through a communication networkusing a communication device, through a centralized server, and soforth).

In some examples, a distance of at least part of the lift (from at leastpart of the second vehicle, from the ground, from an object in theenvironment, from at least part of the first vehicle, from the imagesensor, etc.) may be determined, and the determined state of the liftmay be based on the determined distance of the at least part of thelift. For example, a machine learning model may be trained usingtraining examples to determine distance of parts of lifts by analyzingimages and/or videos, and the trained machine learning model may be usedto determine the distance of the at least part of the lift from the oneor more images. In another example, an artificial neural network (suchas a deep neural networks, convolutional neural networks, etc.) may beconfigured to determine distance of parts of lifts by analyzing imagesand/or videos, and the configured artificial neural network may be usedto determine the distance of the at least part of the lift from the oneor more images. In yet another example, information about the distanceof the at least part of the lift may be received from the second vehicle(for example, using a point to point communication protocol, through acommunication network using a communication device, through acentralized server, and so forth).

In some examples, a motion of at least part of the lift (in relation toat least part of the second vehicle, in relation to the ground, inrelation to the horizon, in relation to an object in the environment, inrelation to the at least part of the first vehicle, in relation to theimage sensor, etc.) may be determined, and the determined state of thelift may be based on the determined motion of the at least part of thelift. For example, a machine learning model may be trained usingtraining examples to determine motion of parts of lifts by analyzingimages and/or videos, and the trained machine learning model may be usedto determine the motion of the at least part of the lift from the one ormore images. In another example, an artificial neural network (such as adeep neural networks, convolutional neural networks, etc.) may beconfigured to determine motion of parts of lifts by analyzing imagesand/or videos, and the configured artificial neural network may be usedto determine the motion of the at least part of the lift from the one ormore images. In yet another example, information about the motion of theat least part of the lift may be received from the second vehicle (forexample, using a point to point communication protocol, through acommunication network using a communication device, through acentralized server, and so forth).

In some embodiments, step 850 may comprise, for example in response to afirst state of the lift determined by step 840, causing the firstvehicle to initiate an action responding to the second vehicle detectedby step 720. Some non-limiting examples of such action may includesignaling, stopping, changing a speed of the first vehicle, changing amotion direction of the first vehicle, passing the second vehicle,forgoing passing the second vehicle, passing the lift, keeping a minimaldistance of at least a selected length from the second vehicle (forexample, the selected length may be less than 100 feet, may be at least100 feet, may be at least 200 feet, etc.), keeping a minimal distance ofat least a selected length from the lift (for example, the selectedlength may be less than 100 feet, may be at least 100 feet, may be atleast 200 feet, etc.), turning, performing a U-turn, driving in reverse,generating an audible warning, and so forth. For example, step 850 maytransmit a signal to an external device (such as a device controllingthe first vehicle, a device navigating the first vehicle, the firstvehicle, a system within the first vehicle, etc.), and the signal may beconfigured to cause the external device to cause the first vehicle toinitiate the action responding to the second vehicle detected by step720. In another example, step 850 may provide information related to thesecond vehicle detected by step 720 (such as position, motion,acceleration, type, dimensions, etc.) to such external device, and theinformation may be configured to cause the external device to cause thefirst vehicle to initiate the action responding to the second vehicledetected by step 720. In one example, in response to a second state ofthe lift determined by step 840, step 850 may cause the first vehicle toinitiate a second action, the second action may differ from the action.

In some examples, step 850 may select the action based on the state ofthe lift determined by step 840. For example, in response to a firststate of the lift determined by step 840, step 850 may select a firstaction, and in response to a second state of the lift determined by step840, step 850 may select a second action (where the second action maydiffer from the first action, and the second state of the lift maydiffer from the first state of the lift). In some examples, in responseto a first state of the lift determined by step 840, step 850 may selecta first action, and in response to a second state of the lift determinedby step 840, step 850 may withhold and/or forgo the first action (forexample, step 850 may withhold and/or forgo causing the first vehicle toinitiate the first action).

In some examples, a motion of the second vehicle may be determined, forexample as described above. Further, in some examples, in response tothe determined motion of the second vehicle and the first determinedstate of the lift, the first vehicle may be caused to initiate theaction (for example as described above). For example, in response to afirst determined motion of the second vehicle and a first determinedstate of the lift, the first vehicle may be caused to initiate a firstaction; in response to a second determined motion of the second vehicleand the first determined state of the lift, the first vehicle may becaused to initiate a second action; and in response to the firstdetermined motion of the second vehicle and a second determined state ofthe lift, the first vehicle may be caused to initiate a third action. Inanother example, in response to a first determined motion of the secondvehicle and a first determined state of the lift, the first vehicle maybe caused to initiate a first action; in response to a second determinedmotion of the second vehicle and the first determined state of the liftand/or in response to the first determined motion of the second vehicleand a second determined state of the lift, causing the first vehicle toinitiate the first action may be withheld and/or forwent.

In some examples, it may be determining that the second vehicle issignaling, for example as described above. Further, in some examples, atype of the signaling of the second vehicle may be determined, forexample as described above. Some non-limiting examples of such type ofsignaling may include signaling using light, signaling using sound,signaling left, signaling right, signaling break, signaling hazardwarning, signaling reversing, signaling warning, and so forth. Further,in some examples, in response to the determination that the secondvehicle is signaling and/or the determined type of the signaling and/orthe first determined state of the lift, the first vehicle may be causedto initiate the action (for example as described above). For example, inresponse to a first determined type of the signaling and a firstdetermined state of the lift, the first vehicle may be caused toinitiate a first action; in response to a second determined type of thesignaling and the first determined state of the lift, the first vehiclemay be caused to initiate a second action; and in response to the firstdetermined type of the signaling and a second determined state of thelift, the first vehicle may be caused to initiate a third action. Inanother example, in response to a first determined type of the signalingand a first determined state of the lift, the first vehicle may becaused to initiate a first action; in response to a second determinedtype of the signaling and the first determined state of the lift and/orin response to the first determined type of the signaling and a seconddetermined state of the lift, causing the first vehicle to initiate thefirst action may be withheld and/or forwent. In an additional example,in response to the determination that the second vehicle is signalingand a first determined state of the lift, the first vehicle may becaused to initiate a first action; and in response to the determinationthat the second vehicle is not signaling and the first determined stateof the lift, the first vehicle may be caused to initiate a secondaction. In yet another example, in response to the determination thatthe second vehicle is signaling and a first determined state of thelift, the first vehicle may be caused to initiate a first action; and inresponse to the determination that the second vehicle is not signalingand the first determined state of the lift, causing the first vehicle toinitiate the first action may be withheld and/or forwent.

In some examples, a position of the second vehicle may be determined,for example as described above. For example, the determined position maybe in relation to the ground, in relation to a map, in relation to aroad, in relation to a lane, in relation to an object in theenvironment, in relation to the at least part of the first vehicle, inrelation to the image sensor, and so forth. Further, in some examples,it may be determining that the second vehicle is in a lane of the firstvehicle, for example as described above. Further, in some examples, itmay be determining that the second vehicle is in a planned path of thefirst vehicle, for example as described above. Further, in someexamples, in response to the determined position of the second vehicleand the first determined state of the lift, the first vehicle may becaused to initiate the action (for example as described above). Forexample, in response to a first determined position of the secondvehicle and a first determined state of the lift, the first vehicle maybe caused to initiate a first action; in response to a second determinedposition of the second vehicle and the first determined state of thelift, the first vehicle may be caused to initiate a second action; andin response to the first determined position of the second vehicle and asecond determined state of the lift, the first vehicle may be caused toinitiate a third action. In another example, in response to a firstdetermined position of the second vehicle and a first determined stateof the lift, the first vehicle may be caused to initiate a first action;in response to a second determined position of the second vehicle andthe first determined state of the lift and/or in response to the firstdetermined position of the second vehicle and a second determined stateof the lift, causing the first vehicle to initiate the first action maybe withheld and/or forwent. In an additional example, in response to thedetermination that the second vehicle is in a lane and/or a planned pathof the first vehicle and a first determined state of the lift, the firstvehicle may be caused to initiate a first action; and in response to thedetermination that the second vehicle is not in a lane and/or is not ina planned path of the first vehicle and the first determined state ofthe lift, the first vehicle may be caused to initiate a second action.In yet another example, in response to the determination that the secondvehicle is in a lane and/or a planned path of the first vehicle and afirst determined state of the lift, the first vehicle may be caused toinitiate a first action; and in response to the determination that thesecond vehicle is not in a lane and/or is not in a planned path of thefirst vehicle and the first determined state of the lift, causing thefirst vehicle to initiate the first action may be withheld and/orforwent.

In some embodiments, systems, methods and computer readable media forcontrolling vehicles in response to outriggers are provided. Onechallenge of autonomous driving is to determine when a standing vehiclehas stopped for a short period of time and therefore the autonomousvehicle needs to wait for the standing vehicle to resume moving, andwhen the standing vehicle has stopped for a longer period of time andtherefore the autonomous vehicle needs to pass the standing vehicle. Insome cases, one indication of whether a vehicle with an outrigger hasstopped for a longer period of time may include the state of theoutrigger. For example, identifying that the outrigger is in use mayindicate that the vehicle with the outrigger has stopped for a longerperiod of time. The provided systems, methods and computer readablemedia for controlling vehicles determine whether the object is carriedby a second vehicle may detect a vehicle connected to a outrigger,identify a state of the outrigger, and control a response of anautonomous vehicle to the detected vehicle based on the identified stateof the outrigger.

FIG. 9 illustrates an example of a method 900 for controlling vehiclesin response to outriggers. In this example, method 900 may comprise:obtaining images captured from an environment of a first vehicle (Step710); analyzing the images to detect a second vehicle (Step 720);analyzing the images to determine that the second vehicle is connectedto a outrigger (Step 930); analyzing the images to determine a state ofthe outrigger (Step 940); in response to a first determined state of theoutrigger, causing the first vehicle to initiate an action responding tothe second vehicle (Step 950), and in response to a second determinedstate of the outrigger, forgoing causing the first vehicle to initiatethe action. In some implementations, method 900 may comprise one or moreadditional steps, while some of the steps listed above may be modifiedor excluded. In some implementations, one or more steps illustrated inFIG. 9 may be executed in a different order and/or one or more groups ofsteps may be executed simultaneously and/or a plurality of steps may becombined into single step and/or a single step may be broken down to aplurality of steps.

In some embodiments, one or more images captured using one or more imagesensors from an environment of a first vehicle may be obtained, forexample as described above. Further, in some examples, the one or moreimages may be analyzed to detect a second vehicle, for example asdescribed above. Further, in some examples, the one or more images maybe analyzed to determine that the second vehicle is connected to anoutrigger. Further, in some examples, the one or more images may beanalyzed to determine a state of the outrigger. Further, in someexamples, in response to a first determined state of the outrigger, thefirst vehicle may be caused to initiate an action responding to thesecond vehicle.

In some embodiments, step 930 may comprise analyzing the one or moreimages obtained by step 710 to determine that the second vehicledetected by step 720 is connected to an outrigger. For example, amachine learning model may be trained using training examples todetermine whether vehicles are connected to outriggers from imagesand/or videos, and step 930 may use the trained machine learning modelto analyze the one or more images obtained by step 710 and determinethat the second vehicle detected by step 720 is connected to anoutrigger. An example of such training example may include an imageand/or a video depicting a vehicle, together with a label indicatingwhether the depicted vehicle is connected to an outrigger. In anotherexample, an artificial neural network (such as a deep neural networks,convolutional neural networks, etc.) may be configured to determinewhether vehicles are connected to outriggers from images and/or videos,and step 930 may use the artificial neural network to analyze the one ormore images obtained by step 710 and determine that the second vehicledetected by step 720 is connected to an outrigger. In yet anotherexample, step 930 may use image classification algorithms to analyze theone or more images obtained by step 710 and determine that the secondvehicle detected by step 720 is connected to an outrigger.

In some embodiments, step 940 may comprise analyzing the one or moreimages obtained by step 710 to determine a state of the outrigger ofstep 930. Some non-limiting examples of such state of the outrigger mayinclude outrigger in use, outrigger not in use, outrigger lifting atleast part of the second vehicle, outrigger not lifting the secondvehicle, outrigger is touching the ground, outrigger is not touching theground, and so forth. For example, a machine learning model may betrained using training examples to determine states of outriggers fromimages and/or videos, and step 940 may use the trained machine learningmodel to analyze the one or more images obtained by step 710 anddetermine the state of the outrigger of step 930. An example of suchtraining example may include an image and/or a video depicting anoutrigger connected to a vehicle, together with a label indicating thestate of the depicted outrigger. In another example, an artificialneural network (such as a deep neural networks, convolutional neuralnetworks, etc.) may be configured to determine states of outriggers fromimages and/or videos, and step 940 may use the artificial neural networkto analyze the one or more images obtained by step 710 and determine thestate of the outrigger of step 930. In yet another example, step 940 mayuse image classification algorithms to analyze the one or more imagesobtained by step 710 and determine the state of the outrigger of step930. Some other non-limiting examples of steps that may be used by step940 for determining the state of the outrigger of step 930 are describedbelow.

In some examples, an orientation of at least part of the outrigger (inrelation to at least part of the second vehicle, in relation to theground, in relation to the horizon, in relation to an object in theenvironment, in relation to the at least part of the first vehicle, inrelation to the image sensor, etc.) may be determined, and thedetermined state of the outrigger may be based on the determinedorientation of the at least part of the outrigger. For example, amachine learning model may be trained using training examples todetermine orientation of parts of outriggers by analyzing images and/orvideos, and the trained machine learning model may be used to determinethe orientation of the at least part of the outrigger from the one ormore images. In another example, an artificial neural network (such as adeep neural networks, convolutional neural networks, etc.) may beconfigured to determine orientation of parts of outriggers by analyzingimages and/or videos, and the configured artificial neural network maybe used to determine the orientation of the at least part of theoutrigger from the one or more images. In yet another example,information about the orientation of the at least part of the outriggermay be received from the second vehicle (for example, using a point topoint communication protocol, through a communication network using acommunication device, through a centralized server, and so forth).

In some examples, a distance of at least part of the outrigger (from atleast part of the second vehicle, from the ground, from an object in theenvironment, from at least part of the first vehicle, from the imagesensor, etc.) may be determined, and the determined state of theoutrigger may be based on the determined distance of the at least partof the outrigger. For example, a machine learning model may be trainedusing training examples to determine distance of parts of outriggers byanalyzing images and/or videos, and the trained machine learning modelmay be used to determine the distance of the at least part of theoutrigger from the one or more images. In another example, an artificialneural network (such as a deep neural networks, convolutional neuralnetworks, etc.) may be configured to determine distance of parts ofoutriggers by analyzing images and/or videos, and the configuredartificial neural network may be used to determine the distance of theat least part of the outrigger from the one or more images. In yetanother example, information about the distance of the at least part ofthe outrigger may be received from the second vehicle (for example,using a point to point communication protocol, through a communicationnetwork using a communication device, through a centralized server, andso forth).

In some examples, a motion of at least part of the outrigger (inrelation to at least part of the second vehicle, in relation to theground, in relation to the horizon, in relation to an object in theenvironment, in relation to the at least part of the first vehicle, inrelation to the image sensor, etc.) may be determined, and thedetermined state of the outrigger may be based on the determined motionof the at least part of the outrigger. For example, a machine learningmodel may be trained using training examples to determine motion ofparts of outriggers by analyzing images and/or videos, and the trainedmachine learning model may be used to determine the motion of the atleast part of the outrigger from the one or more images. In anotherexample, an artificial neural network (such as a deep neural networks,convolutional neural networks, etc.) may be configured to determinemotion of parts of outriggers by analyzing images and/or videos, and theconfigured artificial neural network may be used to determine the motionof the at least part of the outrigger from the one or more images. Inyet another example, information about the motion of the at least partof the outrigger may be received from the second vehicle (for example,using a point to point communication protocol, through a communicationnetwork using a communication device, through a centralized server, andso forth).

In some embodiments, step 950 may comprise, for example in response to afirst state of the outrigger determined by step 940, causing the firstvehicle to initiate an action responding to the second vehicle detectedby step 720. Some non-limiting examples of such action may includesignaling, stopping, changing a speed of the first vehicle, changing amotion direction of the first vehicle, passing the second vehicle,forgoing passing the second vehicle, passing the outrigger, keeping aminimal distance of at least a selected length from the second vehicle(for example, the selected length may be less than 100 feet, may be atleast 100 feet, may be at least 200 feet, etc.), keeping a minimaldistance of at least a selected length from the outrigger (for example,the selected length may be less than 100 feet, may be at least 100 feet,may be at least 200 feet, etc.), turning, performing a U-turn, drivingin reverse, generating an audible warning, and so forth. For example,step 950 may transmit a signal to an external device (such as a devicecontrolling the first vehicle, a device navigating the first vehicle,the first vehicle, a system within the first vehicle, etc.), and thesignal may be configured to cause the external device to cause the firstvehicle to initiate the action responding to the second vehicle detectedby step 720. In another example, step 950 may provide informationrelated to the second vehicle detected by step 720 (such as position,motion, acceleration, type, dimensions, etc.) to such external device,and the information may be configured to cause the external device tocause the first vehicle to initiate the action responding to the secondvehicle detected by step 720. In one example, in response to a secondstate of the outrigger determined by step 940, step 950 may cause thefirst vehicle to initiate a second action, the second action may differfrom the action.

In some examples, step 950 may select the action based on the state ofthe outrigger determined by step 940. For example, in response to afirst state of the outrigger determined by step 940, step 950 may selecta first action, and in response to a second state of the outriggerdetermined by step 940, step 950 may select a second action (where thesecond action may differ from the first action, and the second state ofthe outrigger may differ from the first state of the outrigger). In someexamples, in response to a first state of the outrigger determined bystep 940, step 950 may select a first action, and in response to asecond state of the outrigger determined by step 940, step 950 maywithhold and/or forgo the first action (for example, step 950 maywithhold and/or forgo causing the first vehicle to initiate the action).

In some examples, a motion of the second vehicle may be determined, forexample as described above. Further, in some examples, in response tothe determined motion of the second vehicle and the first determinedstate of the outrigger, the first vehicle may be caused to initiate theaction (for example as described above). For example, in response to afirst determined motion of the second vehicle and a first determinedstate of the outrigger, the first vehicle may be caused to initiate afirst action; in response to a second determined motion of the secondvehicle and the first determined state of the outrigger, the firstvehicle may be caused to initiate a second action; and in response tothe first determined motion of the second vehicle and a seconddetermined state of the outrigger, the first vehicle may be caused toinitiate a third action. In another example, in response to a firstdetermined motion of the second vehicle and a first determined state ofthe outrigger, the first vehicle may be caused to initiate a firstaction; in response to a second determined motion of the second vehicleand the first determined state of the outrigger and/or in response tothe first determined motion of the second vehicle and a seconddetermined state of the outrigger, causing the first vehicle to initiatethe first action may be withheld and/or forwent.

In some examples, it may be determining that the second vehicle issignaling, for example as described above. Further, in some examples, atype of the signaling of the second vehicle may be determined, forexample as described above. Some non-limiting examples of such type ofsignaling may include signaling using light, signaling using sound,signaling left, signaling right, signaling break, signaling hazardwarning, signaling reversing, signaling warning, and so forth. Further,in some examples, in response to the determination that the secondvehicle is signaling and/or the determined type of the signaling and/orthe first determined state of the outrigger, the first vehicle may becaused to initiate the action (for example as described above). Forexample, in response to a first determined type of the signaling and afirst determined state of the outrigger, the first vehicle may be causedto initiate a first action; in response to a second determined type ofthe signaling and the first determined state of the outrigger, the firstvehicle may be caused to initiate a second action; and in response tothe first determined type of the signaling and a second determined stateof the outrigger, the first vehicle may be caused to initiate a thirdaction. In another example, in response to a first determined type ofthe signaling and a first determined state of the outrigger, the firstvehicle may be caused to initiate a first action; in response to asecond determined type of the signaling and the first determined stateof the outrigger and/or in response to the first determined type of thesignaling and a second determined state of the outrigger, causing thefirst vehicle to initiate the first action may be withheld and/orforwent. In an additional example, in response to the determination thatthe second vehicle is signaling and a first determined state of theoutrigger, the first vehicle may be caused to initiate a first action;and in response to the determination that the second vehicle is notsignaling and the first determined state of the outrigger, the firstvehicle may be caused to initiate a second action. In yet anotherexample, in response to the determination that the second vehicle issignaling and a first determined state of the outrigger, the firstvehicle may be caused to initiate a first action; and in response to thedetermination that the second vehicle is not signaling and the firstdetermined state of the outrigger, causing the first vehicle to initiatethe first action may be withheld and/or forwent.

In some examples, a position of the second vehicle may be determined,for example as described above. For example, the determined position maybe in relation to the ground, in relation to a map, in relation to aroad, in relation to a lane, in relation to an object in theenvironment, in relation to the at least part of the first vehicle, inrelation to the image sensor, and so forth. Further, in some examples,it may be determining that the second vehicle is in a lane of the firstvehicle, for example as described above. Further, in some examples, itmay be determining that the second vehicle is in a planned path of thefirst vehicle, for example as described above. Further, in someexamples, in response to the determined position of the second vehicleand the first determined state of the outrigger, the first vehicle maybe caused to initiate the action (for example as described above). Forexample, in response to a first determined position of the secondvehicle and a first determined state of the outrigger, the first vehiclemay be caused to initiate a first action; in response to a seconddetermined position of the second vehicle and the first determined stateof the outrigger, the first vehicle may be caused to initiate a secondaction; and in response to the first determined position of the secondvehicle and a second determined state of the outrigger, the firstvehicle may be caused to initiate a third action. In another example, inresponse to a first determined position of the second vehicle and afirst determined state of the outrigger, the first vehicle may be causedto initiate a first action; in response to a second determined positionof the second vehicle and the first determined state of the outriggerand/or in response to the first determined position of the secondvehicle and a second determined state of the outrigger, causing thefirst vehicle to initiate the first action may be withheld and/orforwent. In an additional example, in response to the determination thatthe second vehicle is in a lane and/or a planned path of the firstvehicle and a first determined state of the outrigger, the first vehiclemay be caused to initiate a first action; and in response to thedetermination that the second vehicle is not in a lane and/or is not ina planned path of the first vehicle and the first determined state ofthe outrigger, the first vehicle may be caused to initiate a secondaction. In yet another example, in response to the determination thatthe second vehicle is in a lane and/or a planned path of the firstvehicle and a first determined state of the outrigger, the first vehiclemay be caused to initiate a first action; and in response to thedetermination that the second vehicle is not in a lane and/or is not ina planned path of the first vehicle and the first determined state ofthe outrigger, causing the first vehicle to initiate the first actionmay be withheld and/or forwent.

In some embodiments, systems, methods and computer readable media forcontrolling vehicles in response to pumps are provided. One challenge ofautonomous driving is to determine when a standing vehicle has stoppedfor a short period of time and therefore the autonomous vehicle needs towait for the standing vehicle to resume moving, and when the standingvehicle has stopped for a longer period of time and therefore theautonomous vehicle needs to pass the standing vehicle. In some cases,one indication of whether a vehicle with a pump has stopped for a longerperiod of time may include the state of the pump. For example,identifying that the pump is in use may indicate that the vehicle withthe pump has stopped for a longer period of time. The provided systems,methods and computer readable media for controlling vehicles may detecta vehicle connected to a pump, identify a state of the pump, and controla response of an autonomous vehicle to the detected vehicle based on theidentified state of the pump.

FIG. 10 illustrates an example of a method for controlling vehicles inresponse to pumps. In this example, method 1000 may comprise: obtainingimages captured from an environment of a first vehicle (Step 710);analyzing the images to detect a second vehicle (Step 720); analyzingthe images to determine that the second vehicle is connected to a pump(Step 1030); analyzing the images to determine a state of the pump (Step1040); in response to a first determined state of the pump, causing thefirst vehicle to initiate an action responding to the second vehicle(Step 1050), and in response to a second determined state of the pump,forgoing causing the first vehicle to initiate the action. In someimplementations, method 1000 may comprise one or more additional steps,while some of the steps listed above may be modified or excluded. Insome implementations, one or more steps illustrated in FIG. 10 may beexecuted in a different order and/or one or more groups of steps may beexecuted simultaneously and/or a plurality of steps may be combined intosingle step and/or a single step may be broken down to a plurality ofsteps.

In some embodiments, one or more images captured using one or more imagesensors from an environment of a first vehicle may be obtained, forexample as described above. Further, in some examples, the one or moreimages may be analyzed to detect a second vehicle, for example asdescribed above. Further, in some examples, the one or more images maybe analyzed to determine that the second vehicle is connected to a pump.Further, in some examples, the one or more images may be analyzed todetermine a state of the pump. Further, in some examples, in response toa first determined state of the pump, the first vehicle may be caused toinitiate an action responding to the second vehicle.

In some embodiments, step 1030 may analyze the one or more imagesobtained by step 710 to determine that the second vehicle detected bystep 720 is connected to a pump. For example, a machine learning modelmay be trained using training examples to determine whether vehicles areconnected to pumps from images and/or videos, and step 1030 may use thetrained machine learning model to analyze the one or more imagesobtained by step 710 and determine that the second vehicle detected bystep 720 is connected to a pump. An example of such training example mayinclude an image and/or a video depicting a vehicle, together with alabel indicating whether the depicted vehicle is connected to a pump. Inanother example, an artificial neural network (such as a deep neuralnetworks, convolutional neural networks, etc.) may be configured todetermine whether vehicles are connected to pumps from images and/orvideos, and step 1030 may use the artificial neural network to analyzethe one or more images obtained by step 710 and determine that thesecond vehicle detected by step 720 is connected to a pump. In yetanother example, step 1030 may use image classification algorithms toanalyze the one or more images obtained by step 710 and determine thatthe second vehicle detected by step 720 is connected to a pump.

In some embodiments, step 1040 may comprise analyzing the one or moreimages obtained by step 710 to determine a state of the pump of Step1030. Some non-limiting examples of such state of the pump may includepump in use, pump not in use, pump connected to ground, pump notconnected to ground, pump connected to an accumulation of liquidsexternal to the second vehicle, pump not connected to an accumulation ofliquids external to the second vehicle, and so forth. For example, amachine learning model may be trained using training examples todetermine states of pumps from images and/or videos, and step 1040 mayuse the trained machine learning model to analyze the one or more imagesobtained by step 710 and determine the state of the pump of Step 1030.An example of such training example may include an image and/or a videodepicting a pump connected to a vehicle, together with a labelindicating the state of the depicted pump. In another example, anartificial neural network (such as a deep neural networks, convolutionalneural networks, etc.) may be configured to determine states of pumpsfrom images and/or videos, and step 1040 may use the artificial neuralnetwork to analyze the one or more images obtained by step 710 anddetermine the state of the pump of Step 1030. In yet another example,step 1040 may use image classification algorithms to analyze the one ormore images obtained by step 710 and determine the state of the pump ofstep 1030. Some other non-limiting examples of steps that may be used bystep 1040 for determining the state of the pump of step 1030 aredescribed below.

In some examples, an orientation of at least part of a pipe (in relationto at least part of the second vehicle, in relation to the ground, inrelation to the horizon, in relation to an object in the environment, inrelation to the at least part of the first vehicle, in relation to theimage sensor, in relation to at least part of the pump, etc.) may bedetermined (wherein the pipe may be connected to the pump), and thedetermined state of the pump may be based on the determined orientationof the at least part of the pipe. For example, a machine learning modelmay be trained using training examples to determine orientation of partsof pipes connected to pumps by analyzing images and/or videos, and thetrained machine learning model may be used to determine the orientationof the at least part of the pipe from the one or more images. In anotherexample, an artificial neural network (such as a deep neural networks,convolutional neural networks, etc.) may be configured to determineorientation of parts of pipes connected to pumps by analyzing imagesand/or videos, and the configured artificial neural network may be usedto determine the orientation of the at least part of the pipe from theone or more images. In yet another example, information about theorientation of the at least part of the pipe may be received from thesecond vehicle (for example, using a point to point communicationprotocol, through a communication network using a communication device,through a centralized server, and so forth).

In some examples, a distance of at least part of a pipe (from at leastpart of the second vehicle, from the ground, from an object in theenvironment, from at least part of the first vehicle, from the imagesensor, from the pump, etc.) may be determined (wherein the pipe may beconnected to the pump), and the determined state of the pump may bebased on the determined distance of the at least part of the pipe. Forexample, a machine learning model may be trained using training examplesto determine distance of parts of pipes connected to pumps by analyzingimages and/or videos, and the trained machine learning model may be usedto determine the distance of the at least part of pipe from the one ormore images. In another example, an artificial neural network (such as adeep neural networks, convolutional neural networks, etc.) may beconfigured to determine distance of parts of pipes connected to pumps byanalyzing images and/or videos, and the configured artificial neuralnetwork may be used to determine the distance of the at least part ofthe pipe from the one or more images. In yet another example,information about the distance of the at least part of the pipe may bereceived from the second vehicle (for example, using a point to pointcommunication protocol, through a communication network using acommunication device, through a centralized server, and so forth).

In some examples, a motion of at least part of a pipe (in relation to atleast part of the second vehicle, in relation to the ground, in relationto the horizon, in relation to an object in the environment, in relationto the at least part of the first vehicle, in relation to the imagesensor, in relation to at least part of the pipe, etc.) may bedetermined (wherein the pipe may be connected to the pump), and thedetermined state of the pump may be based on the determined motion ofthe at least part of the pipe. For example, a machine learning model maybe trained using training examples to determine motion of parts of pipesconnected to pumps by analyzing images and/or videos, and the trainedmachine learning model may be used to determine the motion of the atleast part of the pipe from the one or more images. In another example,an artificial neural network (such as a deep neural networks,convolutional neural networks, etc.) may be configured to determinemotion of parts of pipes connected to pumps by analyzing images and/orvideos, and the configured artificial neural network may be used todetermine the motion of the at least part of the pipe from the one ormore images. In yet another example, information about the motion of theat least part of the pipe may be received from the second vehicle (forexample, using a point to point communication protocol, through acommunication network using a communication device, through acentralized server, and so forth).

In some embodiments, the one or more images obtained by step 710 may beanalyzed to detect a pipe connected to the pump, for example asdescribed below. Further, in some examples, the one or more imagesobtained by step 710 may be analyzed to determine a property of thepipe, for example as described below. Further, in some examples, step1050 may determine the state of the pump based on the determinedproperty of the pipe. For example, step 1050 may use a function thatdetermines a state of the pipe according to the determined property ofthe pipe. In another example, step 1050 may use a lookup table thatcorrelates states of the pipe to properties of the pipe and/or ranges ofvalues of properties of the pipe. In yet another example, in response toa first determined property of the pipe, step 1050 may determine a firststate of the pump, and in response to a second determined property ofthe pipe, step 1050 may determine a second state of the pump, the secondstate of the pump may differ from the first state of the pipe.

In some examples, the one or more images obtained by step 710 may beanalyzed to detect a pipe connected to the pump. For example, a machinelearning model may be trained using training examples to detect pipesconnected to pumps in images and/or videos, and the trained machinelearning model may be used to analyze the one or more images obtained bystep 710 and detect the pipe connected to the pump in the one or moreimages. An example of such training example may include an image and/ora video, together with a label indicating whether a pipe connected tothe pump appears in the image and/or in the video. In another example,an artificial neural network (such as a deep neural networks,convolutional neural networks, etc.) may be configured detect pipesconnected to pumps in images and/or videos, and the configuredartificial neural network may be used to analyze the one or more imagesobtained by step 710 and detect the pipe connected to the pump in theone or more images.

In some examples, the one or more images may be analyzed to determine aproperty of the pipe. For example, a machine learning model may betrained using training examples to determine properties of pipes byanalyzing images and/or videos, and the trained machine learning modelmay be used to analyze the one or more images obtained by step 710 anddetermine the property of the pipe. An example of such training examplemay include an image and/or a video of a pipe, together with a labelindicating properties of the pipe. In another example, an artificialneural network (such as a deep neural networks, convolutional neuralnetworks, etc.) may be configured to determine properties of pipes byanalyzing images and/or videos, and the configured artificial neuralnetwork may be used to analyze the one or more images obtained by step710 and determine the property of the pipe. In yet another example, theproperty of the pipe may include and/or be based on a state of the pipe,and step 1140 may be used to determine the state of the pipe.

In some embodiments, step 1050 may comprise, for example in response toa first determined state of the pump determined by step 1040, causingthe first vehicle to initiate an action responding to the second vehicledetected by step 720. Some non-limiting examples of such action mayinclude signaling, stopping, changing a speed of the first vehicle,changing a motion direction of the first vehicle, passing the secondvehicle, forgoing passing the second vehicle, passing the pump, keepinga minimal distance of at least a selected length from the second vehicle(for example, the selected length may be less than 100 feet, may be atleast 100 feet, may be at least 200 feet, etc.), keeping a minimaldistance of at least a selected length from the pump (for example, theselected length may be less than 100 feet, may be at least 100 feet, maybe at least 200 feet, etc.), turning, performing a U-turn, driving inreverse, generating an audible warning, and so forth. For example, step1050 may transmit a signal to an external device (such as a devicecontrolling the first vehicle, a device navigating the first vehicle,the first vehicle, a system within the first vehicle, etc.), and thesignal may be configured to cause the external device to cause the firstvehicle to initiate the action responding to the second vehicle detectedby step 720. In another example, step 1050 may provide informationrelated to the second vehicle detected by step 720 (such as position,motion, acceleration, type, dimensions, etc.) to such external device,and the information may be configured to cause the external device tocause the first vehicle to initiate the action responding to the secondvehicle detected by step 720. In one example, in response to a secondstate of the pump determined by step 1040, step 1050 may cause the firstvehicle to initiate a second action, the second action may differ fromthe action.

In some examples, step 1050 may select the action based on the state ofthe pump determined by step 1040. For example, in response to a firststate of the pump determined by step 1040, step 1050 may select a firstaction, and in response to a second state of the pump determined by step1040, step 1050 may select a second action (where the second action maydiffer from the first action, and the second state of the pump maydiffer from the first state of the pump). In some examples, in responseto a first state of the pump determined by step 1040, step 1050 mayselect a first action, and in response to a second state of the pumpdetermined by step 1040, step 1050 may withhold and/or forgo the firstaction (for example, step 1050 may withhold and/or forgo causing thefirst vehicle to initiate the action).

In some examples, a motion of the second vehicle may be determined, forexample as described above. Further, in some examples, in response tothe determined motion of the second vehicle and the first determinedstate of the pump, the first vehicle may be caused to initiate theaction (for example as described above). For example, in response to afirst determined motion of the second vehicle and a first determinedstate of the pump, the first vehicle may be caused to initiate a firstaction; in response to a second determined motion of the second vehicleand the first determined state of the pump, the first vehicle may becaused to initiate a second action; and in response to the firstdetermined motion of the second vehicle and a second determined state ofthe pump, the first vehicle may be caused to initiate a third action. Inanother example, in response to a first determined motion of the secondvehicle and a first determined state of the pump, the first vehicle maybe caused to initiate a first action; in response to a second determinedmotion of the second vehicle and the first determined state of the pumpand/or in response to the first determined motion of the second vehicleand a second determined state of the pump, causing the first vehicle toinitiate the first action may be withheld and/or forwent.

In some examples, it may be determining that the second vehicle issignaling, for example as described above. Further, in some examples, atype of the signaling of the second vehicle may be determined, forexample as described above. Some non-limiting examples of such type ofsignaling may include signaling using light, signaling using sound,signaling left, signaling right, signaling break, signaling hazardwarning, signaling reversing, signaling warning, and so forth. Further,in some examples, in response to the determination that the secondvehicle is signaling and/or the determined type of the signaling and/orthe first determined state of the pump, the first vehicle may be causedto initiate the action (for example as described above). For example, inresponse to a first determined type of the signaling and a firstdetermined state of the pump, the first vehicle may be caused toinitiate a first action; in response to a second determined type of thesignaling and the first determined state of the pump, the first vehiclemay be caused to initiate a second action; and in response to the firstdetermined type of the signaling and a second determined state of thepump, the first vehicle may be caused to initiate a third action. Inanother example, in response to a first determined type of the signalingand a first determined state of the pump, the first vehicle may becaused to initiate a first action; in response to a second determinedtype of the signaling and the first determined state of the pump and/orin response to the first determined type of the signaling and a seconddetermined state of the pump, causing the first vehicle to initiate thefirst action may be withheld and/or forwent. In an additional example,in response to the determination that the second vehicle is signalingand a first determined state of the pump, the first vehicle may becaused to initiate a first action; and in response to the determinationthat the second vehicle is not signaling and the first determined stateof the pump, the first vehicle may be caused to initiate a secondaction. In yet another example, in response to the determination thatthe second vehicle is signaling and a first determined state of thepump, the first vehicle may be caused to initiate a first action; and inresponse to the determination that the second vehicle is not signalingand the first determined state of the pump, causing the first vehicle toinitiate the first action may be withheld and/or forwent.

In some examples, a position of the second vehicle may be determined,for example as described above. For example, the determined position maybe in relation to the ground, in relation to a map, in relation to aroad, in relation to a lane, in relation to an object in theenvironment, in relation to the at least part of the first vehicle, inrelation to the image sensor, and so forth. Further, in some examples,it may be determining that the second vehicle is in a lane of the firstvehicle, for example as described above. Further, in some examples, itmay be determining that the second vehicle is in a planned path of thefirst vehicle, for example as described above. Further, in someexamples, in response to the determined position of the second vehicleand the first determined state of the pump, the first vehicle may becaused to initiate the action (for example as described above). Forexample, in response to a first determined position of the secondvehicle and a first determined state of the pump, the first vehicle maybe caused to initiate a first action; in response to a second determinedposition of the second vehicle and the first determined state of thepump, the first vehicle may be caused to initiate a second action; andin response to the first determined position of the second vehicle and asecond determined state of the pump, the first vehicle may be caused toinitiate a third action. In another example, in response to a firstdetermined position of the second vehicle and a first determined stateof the pump, the first vehicle may be caused to initiate a first action;in response to a second determined position of the second vehicle andthe first determined state of the pump and/or in response to the firstdetermined position of the second vehicle and a second determined stateof the pump, causing the first vehicle to initiate the first action maybe withheld and/or forwent. In an additional example, in response to thedetermination that the second vehicle is in a lane and/or a planned pathof the first vehicle and a first determined state of the pump, the firstvehicle may be caused to initiate a first action; and in response to thedetermination that the second vehicle is not in a lane and/or is not ina planned path of the first vehicle and the first determined state ofthe pump, the first vehicle may be caused to initiate a second action.In yet another example, in response to the determination that the secondvehicle is in a lane and/or a planned path of the first vehicle and afirst determined state of the pump, the first vehicle may be caused toinitiate a first action; and in response to the determination that thesecond vehicle is not in a lane and/or is not in a planned path of thefirst vehicle and the first determined state of the pump, causing thefirst vehicle to initiate the first action may be withheld and/orforwent.

In some embodiments, systems, methods and computer readable media forcontrolling vehicles in response to pipes are provided. One challenge ofautonomous driving is to determine when a standing vehicle has stoppedfor a short period of time and therefore the autonomous vehicle needs towait for the standing vehicle to resume moving, and when the standingvehicle has stopped for a longer period of time and therefore theautonomous vehicle needs to pass the standing vehicle. In some cases,one indication of whether a vehicle with a pipe has stopped for a longerperiod of time may include the state of the pipe. For example,identifying that the pipe is in use may indicate that the vehicle withthe pipe has stopped for a longer period of time. The provided systems,methods and computer readable media for controlling vehicles may detecta vehicle connected to a pipe, identify a state of the pipe, and controla response of an autonomous vehicle to the detected vehicle based on theidentified state of the pipe.

FIG. 11 illustrates an example of a method 1100 for controlling vehiclesin response to pipes. In this example, method 1100 may comprise:obtaining images captured from an environment of a first vehicle (Step710); analyzing the images to detect a second vehicle (Step 720);analyzing the images to determine that the second vehicle is connectedto a pipe (Step 1130); analyzing the images to determine a state of thepipe (Step 1140); in response to a first determined state of the pipe,causing the first vehicle to initiate an action responding to the secondvehicle (Step 1150), and in response to a second determined state of thepipe, forgoing causing the first vehicle to initiate the action. In someimplementations, method 1100 may comprise one or more additional steps,while some of the steps listed above may be modified or excluded. Insome implementations, one or more steps illustrated in FIG. 11 may beexecuted in a different order and/or one or more groups of steps may beexecuted simultaneously and/or a plurality of steps may be combined intosingle step and/or a single step may be broken down to a plurality ofsteps.

In some embodiments, one or more images captured using one or more imagesensors from an environment of a first vehicle may be obtained, forexample as described above. Further, in some examples, the one or moreimages may be analyzed to detect a second vehicle, for example asdescribed above. Further, in some examples, the one or more images maybe analyzed to determine that the second vehicle is connected to a pipe.Further, in some examples, the one or more images may be analyzed todetermine a state of the pipe. Further, in some examples, in response toa first determined state of the pipe, the first vehicle may be caused toinitiate an action responding to the second vehicle.

In some embodiments, step 1130 may analyze the one or more imagesobtained by step 710 to determine that the second vehicle detected bystep 720 is connected to a pipe. For example, a machine learning modelmay be trained using training examples to determine whether vehicles areconnected to pipes from images and/or videos, and step 1130 may use thetrained machine learning model to analyze the one or more imagesobtained by step 710 and determine that the second vehicle detected bystep 720 is connected to a pipe. An example of such training example mayinclude an image and/or a video depicting a vehicle, together with alabel indicating whether the depicted vehicle is connected to a pipe. Inanother example, an artificial neural network (such as a deep neuralnetworks, convolutional neural networks, etc.) may be configured todetermine whether vehicles are connected to pipes from images and/orvideos, and step 1130 may use the artificial neural network to analyzethe one or more images obtained by step 710 and determine that thesecond vehicle detected by step 720 is connected to a pipe. In yetanother example, step 1130 may use image classification algorithms toanalyze the one or more images obtained by step 710 and determine thatthe second vehicle detected by step 720 is connected to a pipe.

In some embodiments, step 1140 may comprise analyzing the one or moreimages obtained by step 710 to determine a state of the pipe of step1130. Some non-limiting examples of such state of the pipe may includepipe in use, pipe not in use, pipe touching the ground, pipe nottouching the ground, pipe connected to an accumulation of liquidsexternal to the second vehicle, pipe not connected to an accumulation ofliquids external to the second vehicle, and so forth. For example, amachine learning model may be trained using training examples todetermine states of pipes from images and/or videos, and step 1140 mayuse the trained machine learning model to analyze the one or more imagesobtained by step 710 and determine the state of the pipe of step 1130.An example of such training example may include an image and/or a videodepicting a pipe connected to a vehicle, together with a labelindicating the state of the depicted pipe. In another example, anartificial neural network (such as a deep neural networks, convolutionalneural networks, etc.) may be configured to determine states of pipesfrom images and/or videos, and step 1140 may use the artificial neuralnetwork to analyze the one or more images obtained by step 710 anddetermine the state of the pipe of step 1130. In yet another example,step 1140 may use image classification algorithms to analyze the one ormore images obtained by step 710 and determine the state of the pipe ofstep 1130. Some other non-limiting examples of steps that may be used bystep 1140 for determining the state of the pipe of step 1130 aredescribed below.

In some examples, an orientation of at least part of the pipe (inrelation to at least part of the second vehicle, in relation to theground, in relation to the horizon, in relation to an object in theenvironment, in relation to the at least part of the first vehicle, inrelation to the image sensor, etc.) may be determined, and thedetermined state of the pipe may be based on the determined orientationof the at least part of the pipe. For example, a machine learning modelmay be trained using training examples to determine orientation of partsof pipes by analyzing images and/or videos, and the trained machinelearning model may be used to determine the orientation of the at leastpart of the pipe from the one or more images. In another example, anartificial neural network (such as a deep neural networks, convolutionalneural networks, etc.) may be configured to determine orientation ofparts of pipes by analyzing images and/or videos, and the configuredartificial neural network may be used to determine the orientation ofthe at least part of the pipe from the one or more images. In yetanother example, information about the orientation of the at least partof the pipe may be received from the second vehicle (for example, usinga point to point communication protocol, through a communication networkusing a communication device, through a centralized server, and soforth).

In some examples, a distance of at least part of the pipe (from at leastpart of the second vehicle, from the ground, from an object in theenvironment, from at least part of the first vehicle, from the imagesensor, etc.) may be determined, and the determined state of the pipemay be based on the determined distance of the at least part of thepipe. For example, a machine learning model may be trained usingtraining examples to determine distance of parts of pipes by analyzingimages and/or videos, and the trained machine learning model may be usedto determine the distance of the at least part of the pipe from the oneor more images. In another example, an artificial neural network (suchas a deep neural networks, convolutional neural networks, etc.) may beconfigured to determine distance of parts of pipes by analyzing imagesand/or videos, and the configured artificial neural network may be usedto determine the distance of the at least part of the pipe from the oneor more images. In yet another example, information about the distanceof the at least part of the pipe may be received from the second vehicle(for example, using a point to point communication protocol, through acommunication network using a communication device, through acentralized server, and so forth).

In some examples, a motion of at least part of the pipe (in relation toat least part of the second vehicle, in relation to the ground, inrelation to the horizon, in relation to an object in the environment, inrelation to the at least part of the first vehicle, in relation to theimage sensor, etc.) may be determined, and the determined state of thepipe may be based on the determined motion of the at least part of thepipe. For example, a machine learning model may be trained usingtraining examples to determine motion of parts of pipes by analyzingimages and/or videos, and the trained machine learning model may be usedto determine the motion of the at least part of the pipe from the one ormore images. In another example, an artificial neural network (such as adeep neural networks, convolutional neural networks, etc.) may beconfigured to determine motion of parts of pipes by analyzing imagesand/or videos, and the configured artificial neural network may be usedto determine the motion of the at least part of the pipe from the one ormore images. In yet another example, information about the motion of theat least part of the pipe may be received from the second vehicle (forexample, using a point to point communication protocol, through acommunication network using a communication device, through acentralized server, and so forth).

In some embodiments, step 1150 may comprise, for example in response toa first state of the pipe determined by step 1140, causing the firstvehicle to initiate an action responding to the second vehicle detectedby step 720. Some non-limiting examples of such action may includesignaling, stopping, changing a speed of the first vehicle, changing amotion direction of the first vehicle, passing the second vehicle,forgoing passing the second vehicle, passing the pipe, keeping a minimaldistance of at least a selected length from the second vehicle (forexample, the selected length may be less than 100 feet, may be at least100 feet, may be at least 200 feet, etc.), keeping a minimal distance ofat least a selected length from the pipe (for example, the selectedlength may be less than 100 feet, may be at least 100 feet, may be atleast 200 feet, etc.), turning, performing a U-turn, driving in reverse,generating an audible warning, and so forth. For example, step 1150 maytransmit a signal to an external device (such as a device controllingthe first vehicle, a device navigating the first vehicle, the firstvehicle, a system within the first vehicle, etc.), and the signal may beconfigured to cause the external device to cause the first vehicle toinitiate the action responding to the second vehicle detected by step720. In another example, step 1150 may provide information related tothe second vehicle detected by step 720 to such external device (such asposition, motion, acceleration, type, dimensions, etc.), and theinformation may be configured cause the external device to cause thefirst vehicle to initiate the action responding to the second vehicledetected by step 720.

In some examples, step 1150 may select the action based on the state ofthe pipe determined by step 1140. For example, in response to a firststate of the pipe determined by step 1140, step 1150 may select a firstaction, and in response to a second state of the pipe determined by step1140, step 1150 may select a second action (where the second action maydiffer from the first action, and the second state of the pipe maydiffer from the first state of the pipe). In some examples, in responseto a first state of the pipe determined by step 1140, step 1150 mayselect a first action, and in response to a second state of the pipedetermined by step 1140, step 1150 may withhold and/or forgo the firstaction (for example, step 1150 may withhold and/or forgo causing thefirst vehicle to initiate the action).

In some examples, a motion of the second vehicle may be determined, forexample as described above. Further, in some examples, in response tothe determined motion of the second vehicle and the first determinedstate of the pipe, the first vehicle may be caused to initiate theaction (for example as described above). For example, in response to afirst determined motion of the second vehicle and a first determinedstate of the pipe, the first vehicle may be caused to initiate a firstaction; in response to a second determined motion of the second vehicleand the first determined state of the pipe, the first vehicle may becaused to initiate a second action; and in response to the firstdetermined motion of the second vehicle and a second determined state ofthe pipe, the first vehicle may be caused to initiate a third action. Inanother example, in response to a first determined motion of the secondvehicle and a first determined state of the pipe, the first vehicle maybe caused to initiate a first action; in response to a second determinedmotion of the second vehicle and the first determined state of the pipeand/or in response to the first determined motion of the second vehicleand a second determined state of the pipe, causing the first vehicle toinitiate the first action may be withheld and/or forwent.

In some examples, it may be determining that the second vehicle issignaling, for example as described above. Further, in some examples, atype of the signaling of the second vehicle may be determined, forexample as described above. Some non-limiting examples of such type ofsignaling may include signaling using light, signaling using sound,signaling left, signaling right, signaling break, signaling hazardwarning, signaling reversing, signaling warning, and so forth. Further,in some examples, in response to the determination that the secondvehicle is signaling and/or the determined type of the signaling and/orthe first determined state of the pipe, the first vehicle may be causedto initiate the action (for example as described above). For example, inresponse to a first determined type of the signaling and a firstdetermined state of the pipe, the first vehicle may be caused toinitiate a first action; in response to a second determined type of thesignaling and the first determined state of the pipe, the first vehiclemay be caused to initiate a second action; and in response to the firstdetermined type of the signaling and a second determined state of thepipe, the first vehicle may be caused to initiate a third action. Inanother example, in response to a first determined type of the signalingand a first determined state of the pipe, the first vehicle may becaused to initiate a first action; in response to a second determinedtype of the signaling and the first determined state of the pipe and/orin response to the first determined type of the signaling and a seconddetermined state of the pipe, causing the first vehicle to initiate thefirst action may be withheld and/or forwent. In an additional example,in response to the determination that the second vehicle is signalingand a first determined state of the pipe, the first vehicle may becaused to initiate a first action; and in response to the determinationthat the second vehicle is not signaling and the first determined stateof the pipe, the first vehicle may be caused to initiate a secondaction. In yet another example, in response to the determination thatthe second vehicle is signaling and a first determined state of thepipe, the first vehicle may be caused to initiate a first action; and inresponse to the determination that the second vehicle is not signalingand the first determined state of the pipe, causing the first vehicle toinitiate the first action may be withheld and/or forwent.

In some examples, a position of the second vehicle may be determined,for example as described above. For example, the determined position maybe in relation to the ground, in relation to a map, in relation to aroad, in relation to a lane, in relation to an object in theenvironment, in relation to the at least part of the first vehicle, inrelation to the image sensor, and so forth. Further, in some examples,it may be determining that the second vehicle is in a lane of the firstvehicle, for example as described above. Further, in some examples, itmay be determining that the second vehicle is in a planned path of thefirst vehicle, for example as described above. Further, in someexamples, in response to the determined position of the second vehicleand the first determined state of the pipe, the first vehicle may becaused to initiate the action (for example as described above). Forexample, in response to a first determined position of the secondvehicle and a first determined state of the pipe, the first vehicle maybe caused to initiate a first action; in response to a second determinedposition of the second vehicle and the first determined state of thepipe, the first vehicle may be caused to initiate a second action; andin response to the first determined position of the second vehicle and asecond determined state of the pipe, the first vehicle may be caused toinitiate a third action. In another example, in response to a firstdetermined position of the second vehicle and a first determined stateof the pipe, the first vehicle may be caused to initiate a first action;in response to a second determined position of the second vehicle andthe first determined state of the pipe and/or in response to the firstdetermined position of the second vehicle and a second determined stateof the pipe, causing the first vehicle to initiate the first action maybe withheld and/or forwent. In an additional example, in response to thedetermination that the second vehicle is in a lane and/or a planned pathof the first vehicle and a first determined state of the pipe, the firstvehicle may be caused to initiate a first action; and in response to thedetermination that the second vehicle is not in a lane and/or is not ina planned path of the first vehicle and the first determined state ofthe pipe, the first vehicle may be caused to initiate a second action.In yet another example, in response to the determination that the secondvehicle is in a lane and/or a planned path of the first vehicle and afirst determined state of the pipe, the first vehicle may be caused toinitiate a first action; and in response to the determination that thesecond vehicle is not in a lane and/or is not in a planned path of thefirst vehicle and the first determined state of the pipe, causing thefirst vehicle to initiate the first action may be withheld and/orforwent.

In some embodiments, systems, methods and computer readable media forcontrolling vehicles in response to objects are provided. One challengeof autonomous driving is to determine when an object has stopped for ashort period of time and therefore the autonomous vehicle needs to waitfor the object to move (such as a person or a an animal standing on theroad, an object being carried by a vehicle, and so forth), and when theobject has stopped for a longer period of time and therefore theautonomous vehicle needs to pass the object (such as a dead animal onthe road, a rock on the road, and so forth). In some cases, identifyingthat the object is in being carried by a vehicle may indicate that theobject has stopped for a shorter period of time. The provided systems,methods and computer readable media for controlling vehicles may detectan object, determine whether the object is carried by a vehicle, andcontrol a response of an autonomous vehicle to the detected object basedon the determination.

FIG. 12 illustrates an example of a method 1200 for controlling vehiclesin response to objects. In this example, method 1200 may comprise:obtaining images captured from an environment of a first vehicle (Step710); analyzing the images to detect an object (Step 1220); analyzingthe images to determine whether the object is carried by a secondvehicle (Step 1230); in response to a determination that the object isnot carried by a second vehicle, causing the first vehicle to initiatean action responding to the object (Step 1240); and in response to adetermination that the object is carried by a second vehicle, forgoingcausing the first vehicle to initiate the action (Step 1250). Somenon-limiting examples of such object may include a third vehicle, a roadroller, a tractor, heavy equipment, a scrap, a dead animal, and soforth. In some implementations, method 1200 may comprise one or moreadditional steps, while some of the steps listed above may be modifiedor excluded. In some implementations, one or more steps illustrated inFIG. 12 may be executed in a different order and/or one or more groupsof steps may be executed simultaneously and/or a plurality of steps maybe combined into single step and/or a single step may be broken down toa plurality of steps.

In some embodiments, one or more images captured using one or more imagesensors from an environment of a first vehicle may be obtained, forexample as described above. Further, in some examples, the one or moreimages may be analyzed to detect an object. Further, in some examples,it may be determined whether the object is carried by a second vehicle(for example, the object may be towed by the second vehicle, the objectis loaded onto the second vehicle, and so forth). Further, in someexamples, for example in response to the determination that the objectis not carried by a second vehicle, the first vehicle may be caused toinitiate an action responding to the object. Further, in some examples,for example in response to the determination that the object is carriedby a second vehicle, causing the first vehicle to initiate the actionmay be withheld and/or forwent. Some non-limiting examples of suchaction may include signaling, stopping, changing a speed of the firstvehicle, changing a motion direction of the first vehicle, passing theobject, forgoing passing the object, keeping a minimal distance of atleast a selected length from the object (for example, the selectedlength may be less than 100 feet, may be at least 100 feet, may be atleast 200 feet, etc.), turning, performing a U-turn, driving in reverse,generating an audible warning, and so forth. Some non-limiting examplesof such object may include a third vehicle, a road roller, a tractor,heavy equipment, a scrap, a dead animal, and so forth.

In some embodiments, step 1220 may comprise analyzing the one or moreimages obtained by step 710 to detect an object (for example, to detectan object of particular types, an object of particular dimensions, anobject of any type, and so forth). Some non-limiting examples of suchobject may include a third vehicle, a road roller, a tractor, heavyequipment, a scrap, an animal, a dead animal, and so forth. For example,a machine learning model may be trained using training examples todetect objects (of particular types, of particular dimensions, of anytype, etc.) in images and/or videos, and step 1220 may use the trainedmachine learning model to analyze the one or more images obtained bystep 710 and detect the object in the one or more images. An example ofsuch training example may include an image and/or a video, together witha label indicating whether an object (of particular types, of particulardimensions, of any type, etc.) appears in the image and/or in the video,and/or a position of a depiction of the object in the image and/or inthe video. In another example, an artificial neural network (such as adeep neural networks, convolutional neural networks, etc.) may beconfigured detect objects (of particular types, of particulardimensions, of any type, etc.) in images and/or videos, and step 1220may use the configured artificial neural network to analyze the one ormore images obtained by step 710 and detect the object in the one ormore images. In yet another example, step 1220 may use object detectionalgorithms configured to detect objects (of particular types, ofparticular dimensions, of any type, etc.) in images and/or videos todetect the object in the one or more images.

In some embodiments, step 1230 may comprise analyzing the one or moreimages obtained by step 710 to determine whether the object detected bystep 1220 is carried by a second vehicle. For example, a machinelearning model may be trained using training examples to determinewhether objects are carried by vehicles by analyzing images and/orvideos, and step 1230 may use the trained machine learning model toanalyze the one or more images obtained by step 710 and determinewhether the object detected by step 1220 is carried by a second vehicle.An example of such training example may include an image and/or a videodepicting a particular object (possibly with an indication of theparticular object), together with a label indicating whether theparticular object is carried by a vehicle. In another example, anartificial neural network (such as a deep neural networks, convolutionalneural networks, etc.) may be configured to determine whether objectsare carried by vehicles by analyzing images and/or videos, and step 1230may use the configured artificial neural network to analyze the one ormore images obtained by step 710 and determine whether the objectdetected by step 1220 is carried by a second vehicle. In yet anotherexample, step 1230 may use object detection algorithms to attempt todetect a vehicle in the one or more images obtained by step 710, maydetermine that the object detected by step 1220 is carried by a secondvehicle in response to a detection of a vehicle under the objectdetected by step 1220, and may determine that the object detected bystep 1220 is not carried by a second vehicle in response to no detectionof a vehicle under the object detected by step 1220. In an additionalexample, step 1230 may use motion detection algorithms to attempt todetermine in the one or more images obtained by step 710 a motion of avehicle and a motion of the object detected by step 1220, may determinethat the object detected by step 1220 is carried by a second vehicle inresponse to the two determined motions being correlated over time (i.e.,the vehicle and the object moving in the same speed and direction), andmay determine that the object detected by step 1220 is not carried by asecond vehicle in response to the two determined motions beinguncorrelated over time (i.e., the vehicle and the object not moving inthe same speed and direction).

In some examples, step 1230 may comprise analyzing the one or moreimages obtained by step 710 to determine whether the object detected bystep 1220 is towed by a second vehicle, and step 1230 may determinewhether the object detected by step 1220 is carried by a second vehiclebased on the determination of whether the object detected by step 1220is towed by a second vehicle. For example, in response to adetermination that the object detected by step 1220 is being towed by asecond vehicle, step 1230 may determine that the object detected by step1220 is carried by a second vehicle. In one example, in response adetermination that the object detected by step 1220 is not being towedby a second vehicle, step 1230 may determine that the object detected bystep 1220 is not carried by a second vehicle. In another example, inresponse a determination that the object detected by step 1220 is notbeing towed by a second vehicle, step 1230 may conduct additional stepsto determine whether the object detected by step 1220 is carried by asecond vehicle. In one example, a machine learning model may be trainedusing training examples to determine whether objects are towed byvehicles by analyzing images and/or videos, and step 1230 may use thetrained machine learning model to analyze the one or more imagesobtained by step 710 and determine whether the object detected by step1220 is towed by a second vehicle. An example of such training examplemay include an image and/or a video depicting a particular object(possibly with an indication of the particular object), together with alabel indicating whether the particular object is towed by a vehicle. Inanother example, an artificial neural network (such as a deep neuralnetworks, convolutional neural networks, etc.) may be configured todetermine whether objects are towed by vehicles by analyzing imagesand/or videos, and step 1230 may use the configured artificial neuralnetwork to analyze the one or more images obtained by step 710 anddetermine whether the object detected by step 1220 is towed by a secondvehicle.

In some examples, step 1230 may comprise analyzing the one or moreimages obtained by step 710 to determine whether the object detected bystep 1220 is loaded onto a second vehicle, and step 1230 may determinewhether the object detected by step 1220 is carried by a second vehiclebased on the determination of whether the object detected by step 1220is loaded onto a second vehicle. For example, in response to adetermination that the object detected by step 1220 is being loaded ontoa second vehicle, step 1230 may determine that the object detected bystep 1220 is carried by a second vehicle. In one example, in response adetermination that the object detected by step 1220 is not being loadedonto a second vehicle, step 1230 may determine that the object detectedby step 1220 is not carried by a second vehicle. In another example, inresponse a determination that the object detected by step 1220 is notbeing loaded onto a second vehicle, step 1230 may conduct additionalsteps to determine whether the object detected by step 1220 is carriedby a second vehicle. In one example, a machine learning model may betrained using training examples to determine whether objects are loadedonto vehicles by analyzing images and/or videos, and step 1230 may usethe trained machine learning model to analyze the one or more imagesobtained by step 710 and determine whether the object detected by step1220 is loaded onto a second vehicle. An example of such trainingexample may include an image and/or a video depicting a particularobject (possibly with an indication of the particular object), togetherwith a label indicating whether the particular object is loaded onto avehicle. In another example, an artificial neural network (such as adeep neural networks, convolutional neural networks, etc.) may beconfigured to determine whether objects are loaded onto vehicles byanalyzing images and/or videos, and step 1230 may use the configuredartificial neural network to analyze the one or more images obtained bystep 710 and determine whether the object detected by step 1220 isloaded onto a second vehicle.

Additionally or alternatively, indication that the second vehicle iscarrying the object and/or information about the object carried by thesecond vehicle may be received from the second vehicle (for example,using a point to point communication protocol, through a communicationnetwork using a communication device, through a centralized server, andso forth), and step 1230 may determine that the object is carried by thesecond vehicle based on the received indication and/or on the receivedinformation.

In some embodiments, step 1240 may comprise, for example in response toa determination by step 1230 that the object is not carried by a secondvehicle, causing the first vehicle to initiate an action responding tothe object. Some non-limiting examples of such action may includesignaling, stopping, changing a speed of the first vehicle, changing amotion direction of the first vehicle, passing the object, forgoingpassing the object, keeping a minimal distance of at least a selectedlength from the object (for example, the selected length may be lessthan 100 feet, may be at least 100 feet, may be at least 200 feet,etc.), turning, performing a U-turn, driving in reverse, generating anaudible warning, and so forth. Further, in some examples, in response tothe determination that the object is carried by a second vehicle, step1240 (and step 1250) may withhold and/or forgo causing the first vehicleto initiate the action. In one example, step 1240 may transmit a signalto an external device (such as a device controlling the first vehicle, adevice navigating the first vehicle, the first vehicle, a system withinthe first vehicle, etc.), and the signal may be configured to cause theexternal device to cause the first vehicle to initiate the actionresponding to the object. In another example, step 1240 may provideinformation related to the object (such as position, type, dimensions,etc.) to such external device, and the information may be configured tocause the external device to cause the first vehicle to initiate theaction responding to the object.

Additionally or alternatively, in response to a determination by step1230 that the object is carried by a second vehicle, step 1250 may causethe first vehicle to initiate a different action responding to theobject, for example by transmitting a signal and/or by providinginformation as described above in relation to step 1240.

In some examples, step 1220 may analyze the one or more images obtainedby step 710 to detect an animal (for example as described above), andstep 1230 may analyze the one or more images obtained by step 710 todetermine whether the animal detected by step 1220 is carried by asecond vehicle (for example as described above). Further, in oneexample, in response to a determination by step 1230 that the animaldetected by step 1220 is not carried by a second vehicle, step 1240 maycause the first vehicle to initiate an action responding to the animaldetected by step 1220 (for example as described above), and in responseto a determination by step 1230 that the object is carried by a secondvehicle, step 1250 may withhold and/or forgo causing the first vehicleto initiate the action.

In some examples, step 1220 may analyze the one or more images obtainedby step 710 to detect a dead animal (for example as described above),and step 1230 may analyze the one or more images obtained by step 710 todetermine whether the dead animal detected by step 1220 is carried by asecond vehicle (for example as described above). Further, in oneexample, in response to a determination by step 1230 that the deadanimal detected by step 1220 is not carried by a second vehicle, step1240 may cause the first vehicle to initiate an action responding to thedead animal detected by step 1220 (for example as described above), andin response to a determination by step 1230 that the object is carriedby a second vehicle, step 1250 may withhold and/or forgo causing thefirst vehicle to initiate the action.

In some examples, step 1220 may analyze the one or more images obtainedby step 710 to detect a scrap (for example as described above), and step1230 may analyze the one or more images obtained by step 710 todetermine whether the scrap detected by step 1220 is carried by a secondvehicle (for example as described above). Further, in one example, inresponse to a determination by step 1230 that the scrap detected by step1220 is not carried by a second vehicle, step 1240 may cause the firstvehicle to initiate an action responding to the scrap detected by step1220 (for example as described above), and in response to adetermination by step 1230 that the object is carried by a secondvehicle, step 1250 may withhold and/or forgo causing the first vehicleto initiate the action.

In some examples, step 1220 may analyze the one or more images obtainedby step 710 to detect a heavy equipment (for example as describedabove), and step 1230 may analyze the one or more images obtained bystep 710 to determine whether the heavy equipment detected by step 1220is carried by a second vehicle (for example as described above).Further, in one example, in response to a determination by step 1230that the heavy equipment detected by step 1220 is not carried by asecond vehicle, step 1240 may cause the first vehicle to initiate anaction responding to the heavy equipment detected by step 1220 (forexample as described above), and in response to a determination by step1230 that the object is carried by a second vehicle, step 1250 maywithhold and/or forgo causing the first vehicle to initiate the action.

In some examples, step 1220 may analyze the one or more images obtainedby step 710 to detect a tractor (for example as described above), andstep 1230 may analyze the one or more images obtained by step 710 todetermine whether the tractor detected by step 1220 is carried by asecond vehicle (for example as described above). Further, in oneexample, in response to a determination by step 1230 that the tractordetected by step 1220 is not carried by a second vehicle, step 1240 maycause the first vehicle to initiate an action responding to the tractordetected by step 1220 (for example as described above), and in responseto a determination by step 1230 that the object is carried by a secondvehicle, step 1250 may withhold and/or forgo causing the first vehicleto initiate the action.

In some examples, step 1220 may analyze the one or more images obtainedby step 710 to detect a road roller (for example as described above),and step 1230 may analyze the one or more images obtained by step 710 todetermine whether the road roller detected by step 1220 is carried by asecond vehicle (for example as described above). Further, in oneexample, in response to a determination by step 1230 that the roadroller detected by step 1220 is not carried by a second vehicle, step1240 may cause the first vehicle to initiate an action responding to theroad roller detected by step 1220 (for example as described above), andin response to a determination by step 1230 that the object is carriedby a second vehicle, step 1250 may withhold and/or forgo causing thefirst vehicle to initiate the action.

In some examples, step 1220 may analyze the one or more images obtainedby step 710 to detect a third vehicle (for example as described above),and step 1230 may analyze the one or more images obtained by step 710 todetermine whether the third vehicle detected by step 1220 is carried bya second vehicle (for example as described above). Further, in oneexample, in response to a determination by step 1230 that the thirdvehicle detected by step 1220 is not carried by a second vehicle, step1240 may cause the first vehicle to initiate an action responding to thethird vehicle detected by step 1220 (for example as described above),and in response to a determination by step 1230 that the object iscarried by a second vehicle, step 1250 may withhold and/or forgo causingthe first vehicle to initiate the action.

In some embodiments, systems, methods and computer readable media forcontrolling vehicles based on doors of other vehicles are provided. Onechallenge of autonomous driving is to determine when a standing vehiclehas stopped for a short period of time and therefore the autonomousvehicle needs to wait for the standing vehicle to resume moving, andwhen the standing vehicle has stopped for a longer period of time andtherefore the autonomous vehicle needs to pass the standing vehicle. Insome cases, one indication of whether a vehicle with a door has stoppedfor a longer period of time may include the state of the door. Forexample, identifying that the door is open may indicate that the vehiclewith the door has stopped for a longer period of time. The providedsystems, methods and computer readable media for controlling vehiclesmay detect a vehicle connected to a door, identify a state of the door,and control a response of an autonomous vehicle to the detected vehiclebased on the identified state of the door.

FIG. 13 illustrates an example of a method 1300 for controlling vehiclesbased on doors of other vehicles. In this example, method 1300 maycomprise: obtaining images captured from an environment of a firstvehicle (Step 710); analyzing the images to detect a second vehicle(Step 720); analyzing the images to determine a state of a door of thesecond vehicle (Step 1330); in response to a first determined state ofthe door, causing the first vehicle to initiate an action responding tothe second vehicle (Step 1340), and in response to a second determinedstate of the door, forgoing causing the first vehicle to initiate theaction. In some implementations, method 1300 may comprise one or moreadditional steps, while some of the steps listed above may be modifiedor excluded. In some implementations, one or more steps illustrated inFIG. 13 may be executed in a different order and/or one or more groupsof steps may be executed simultaneously and/or a plurality of steps maybe combined into single step and/or a single step may be broken down toa plurality of steps.

In some embodiments, one or more images captured using one or more imagesensors from an environment of a first vehicle may be obtained, forexample as described above. Further, in some examples, the one or moreimages may be analyzed to detect a second vehicle, for example asdescribed above. Further, in some examples, the one or more images maybe analyzed to determine a state of a door of the second vehicle.Further, in some examples, in response to a first determined state ofthe door of the second vehicle, the first vehicle may be caused toinitiate an action responding to the second vehicle.

In some embodiments, step 1330 may comprise analyzing the one or moreimages obtained by step 710 to determine a state of the door of thesecond vehicle detected by step 720. Some non-limiting examples of suchstate of the door of the second vehicle may include door of the secondvehicle in open, door of the second vehicle not in close, door of thesecond vehicle completely open, door of the second vehicle substantiallyclosed, door of the second vehicle in opening, door of the secondvehicle not in closing, and so forth. For example, a machine learningmodel may be trained using training examples to determine states ofdoors from images and/or videos, and step 1330 may use the trainedmachine learning model to analyze the one or more images obtained bystep 710 and determine the state of the door of the second vehicledetected by step 720. An example of such training example may include animage and/or a video depicting a vehicle, together with a labelindicating the state of a door of the depicted vehicle. In anotherexample, an artificial neural network (such as a deep neural networks,convolutional neural networks, etc.) may be configured to determinestates of doors from images and/or videos, and step 1330 may use theartificial neural network to analyze the one or more images obtained bystep 710 and determine the state of the door of the second vehicledetected by step 720. In yet another example, step 1330 may use imageclassification algorithms to analyze the one or more images obtained bystep 710 and determine the state of the door of the second vehicledetected by step 720. Some other non-limiting examples of steps that maybe used by step 1330 for determining the state of the door of the secondvehicle are described below.

In some examples, an orientation of at least part of the door of thesecond vehicle (in relation to at least one other part of the secondvehicle, in relation to the ground, in relation to the horizon, inrelation to an object in the environment, in relation to the at leastpart of the first vehicle, in relation to the image sensor, etc.) may bedetermined, and the determined state of the door of the second vehiclemay be based on the determined orientation of the at least part of thedoor of the second vehicle. For example, a machine learning model may betrained using training examples to determine orientation of parts ofdoors by analyzing images and/or videos, and the trained machinelearning model may be used to determine the orientation of the at leastpart of the door of the second vehicle from the one or more images. Inanother example, an artificial neural network (such as a deep neuralnetworks, convolutional neural networks, etc.) may be configured todetermine orientation of parts of doors by analyzing images and/orvideos, and the configured artificial neural network may be used todetermine the orientation of the at least part of the door of the secondvehicle from the one or more images. In yet another example, informationabout the orientation of the at least part of the door of the secondvehicle may be received from the second vehicle (for example, using apoint to point communication protocol, through a communication networkusing a communication device, through a centralized server, and soforth).

In some examples, a distance of at least part of the door of the secondvehicle (from at least part of the second vehicle, from the ground, froman object in the environment, from at least part of the first vehicle,from the image sensor, etc.) may be determined, and the determined stateof the door of the second vehicle may be based on the determineddistance of the at least part of the door of the second vehicle. Forexample, a machine learning model may be trained using training examplesto determine distance of parts of doors by analyzing images and/orvideos, and the trained machine learning model may be used to determinethe distance of the at least part of the door of the second vehicle fromthe one or more images. In another example, an artificial neural network(such as a deep neural networks, convolutional neural networks, etc.)may be configured to determine distance of parts of doors by analyzingimages and/or videos, and the configured artificial neural network maybe used to determine the distance of the at least part of the door ofthe second vehicle from the one or more images. In yet another example,information about the distance of the at least part of the door of thesecond vehicle may be received from the second vehicle (for example,using a point to point communication protocol, through a communicationnetwork using a communication device, through a centralized server, andso forth).

In some examples, a motion of at least part of the door of the secondvehicle (in relation to at least one other part of the second vehicle,in relation to the ground, in relation to the horizon, in relation to anobject in the environment, in relation to the at least part of the firstvehicle, in relation to the image sensor, etc.) may be determined, andthe determined state of the door of the second vehicle may be based onthe determined motion of the at least part of the door of the secondvehicle. For example, a machine learning model may be trained usingtraining examples to determine motion of parts of doors by analyzingimages and/or videos, and the trained machine learning model may be usedto determine the motion of the at least part of the door of the secondvehicle from the one or more images. In another example, an artificialneural network (such as a deep neural networks, convolutional neuralnetworks, etc.) may be configured to determine motion of parts of doorsby analyzing images and/or videos, and the configured artificial neuralnetwork may be used to determine the motion of the at least part of thedoor of the second vehicle from the one or more images. In yet anotherexample, information about the motion of the at least part of the doorof the second vehicle may be received from the second vehicle (forexample, using a point to point communication protocol, through acommunication network using a communication device, through acentralized server, and so forth).

In some embodiments, step 1340 may comprise, for example in response toa first state of the door of the second vehicle detected by step 720(e.g., as determined by step 1330), causing the first vehicle toinitiate an action responding to the second vehicle. Some non-limitingexamples of such action may include signaling, stopping, changing aspeed of the first vehicle, changing a motion direction of the firstvehicle, passing the second vehicle, forgoing passing the secondvehicle, passing the door, keeping a minimal distance of at least aselected length from the second vehicle (for example, the selectedlength may be less than 10 feet, may be at least 10 feet, may be atleast 20 feet, etc.), keeping a minimal distance of at least a selectedlength from the door (for example, the selected length may be less than10 feet, may be at least 10 feet, may be at least 20 feet, etc.),turning, performing a U-turn, driving in reverse, generating an audiblewarning, and so forth. For example, step 1340 may transmit a signal toan external device (such as a device controlling the first vehicle, adevice navigating the first vehicle, the first vehicle, a system withinthe first vehicle, etc.), and the signal may be configured to cause theexternal device to cause the first vehicle to initiate the actionresponding to the second vehicle detected by step 720. In anotherexample, step 1340 may provide information related to the second vehicledetected by step 720 (such as position, motion, acceleration, type,dimensions, etc.) to such external device, and the information may beconfigured to cause the external device to cause the first vehicle toinitiate the action responding to the second vehicle detected by step720. In one example, in response to a second state of the door of thesecond vehicle detected by step 720 (e.g., as determined by step 1330),step 1340 may cause the first vehicle to initiate a second action, thesecond action may differ from the action.

In some examples, step 1340 may select the action based on the state ofthe door of the second vehicle detected by step 720 (e.g., as determinedby step 1330). For example, in response to a first state of the door ofthe second vehicle detected by step 720 (e.g., as determined by step1330), step 1340 may select a first action, and in response to a secondstate of the door of the second vehicle detected by step 720 (e.g., asdetermined by step 1330), step 1340 may select a second action (wherethe second action may differ from the first action, and the second stateof the door of the second vehicle may differ from the first state of thedoor of the second vehicle). In some examples, in response to a firststate of the door of the second vehicle detected by step 720 (e.g., asdetermined by step 1330), step 1340 may select a first action, and inresponse to a second state of the door of the second vehicle detected bystep 720 (e.g., as determined by step 1330), step 1340 may withholdand/or forgo the first action (for example, step 1340 may withholdand/or forgo causing the first vehicle to initiate the action).

In some examples, a motion of the second vehicle may be determined, forexample as described above. Further, in some examples, in response tothe determined motion of the second vehicle and the first determinedstate of the door of the second vehicle, the first vehicle may be causedto initiate the action (for example as described above). For example, inresponse to a first determined motion of the second vehicle and a firstdetermined state of the door of the second vehicle, the first vehiclemay be caused to initiate a first action; in response to a seconddetermined motion of the second vehicle and the first determined stateof the door of the second vehicle, the first vehicle may be caused toinitiate a second action; and in response to the first determined motionof the second vehicle and a second determined state of the door of thesecond vehicle, the first vehicle may be caused to initiate a thirdaction. In another example, in response to a first determined motion ofthe second vehicle and a first determined state of the door of thesecond vehicle, the first vehicle may be caused to initiate a firstaction; in response to a second determined motion of the second vehicleand the first determined state of the door of the second vehicle and/orin response to the first determined motion of the second vehicle and asecond determined state of the door of the second vehicle, causing thefirst vehicle to initiate the first action may be withheld and/orforwent.

In some examples, it may be determining that the second vehicle issignaling, for example as described above. Further, in some examples, atype of the signaling of the second vehicle may be determined, forexample as described above. Some non-limiting examples of such type ofsignaling may include signaling using light, signaling using sound,signaling left, signaling right, signaling break, signaling hazardwarning, signaling reversing, signaling warning, and so forth. Further,in some examples, in response to the determination that the secondvehicle is signaling and/or the determined type of the signaling and/orthe first determined state of the door of the second vehicle, the firstvehicle may be caused to initiate the action (for example as describedabove). For example, in response to a first determined type of thesignaling and a first determined state of the door of the secondvehicle, the first vehicle may be caused to initiate a first action; inresponse to a second determined type of the signaling and the firstdetermined state of the door of the second vehicle, the first vehiclemay be caused to initiate a second action; and in response to the firstdetermined type of the signaling and a second determined state of thedoor of the second vehicle, the first vehicle may be caused to initiatea third action. In another example, in response to a first determinedtype of the signaling and a first determined state of the door of thesecond vehicle, the first vehicle may be caused to initiate a firstaction; in response to a second determined type of the signaling and thefirst determined state of the door of the second vehicle and/or inresponse to the first determined type of the signaling and a seconddetermined state of the door of the second vehicle, causing the firstvehicle to initiate the first action may be withheld and/or forwent. Inan additional example, in response to the determination that the secondvehicle is signaling and a first determined state of the door of thesecond vehicle, the first vehicle may be caused to initiate a firstaction; and in response to the determination that the second vehicle isnot signaling and the first determined state of the door of the secondvehicle, the first vehicle may be caused to initiate a second action. Inyet another example, in response to the determination that the secondvehicle is signaling and a first determined state of the door of thesecond vehicle, the first vehicle may be caused to initiate a firstaction; and in response to the determination that the second vehicle isnot signaling and the first determined state of the door of the secondvehicle, causing the first vehicle to initiate the first action may bewithheld and/or forwent.

In some examples, a position of the second vehicle may be determined,for example as described above. For example, the determined position maybe in relation to the ground, in relation to a map, in relation to aroad, in relation to a lane, in relation to an object in theenvironment, in relation to the at least part of the first vehicle, inrelation to the image sensor, and so forth. Further, in some examples,it may be determining that the second vehicle is in a lane of the firstvehicle, for example as described above. Further, in some examples, itmay be determining that the second vehicle is in a planned path of thefirst vehicle, for example as described above. Further, in someexamples, in response to the determined position of the second vehicleand the first determined state of the door of the second vehicle, thefirst vehicle may be caused to initiate the action (for example asdescribed above). For example, in response to a first determinedposition of the second vehicle and a first determined state of the doorof the second vehicle, the first vehicle may be caused to initiate afirst action; in response to a second determined position of the secondvehicle and the first determined state of the door of the secondvehicle, the first vehicle may be caused to initiate a second action;and in response to the first determined position of the second vehicleand a second determined state of the door of the second vehicle, thefirst vehicle may be caused to initiate a third action. In anotherexample, in response to a first determined position of the secondvehicle and a first determined state of the door of the second vehicle,the first vehicle may be caused to initiate a first action; in responseto a second determined position of the second vehicle and the firstdetermined state of the door of the second vehicle and/or in response tothe first determined position of the second vehicle and a seconddetermined state of the door of the second vehicle, causing the firstvehicle to initiate the first action may be withheld and/or forwent. Inan additional example, in response to the determination that the secondvehicle is in a lane and/or a planned path of the first vehicle and afirst determined state of the door of the second vehicle, the firstvehicle may be caused to initiate a first action; and in response to thedetermination that the second vehicle is not in a lane and/or is not ina planned path of the first vehicle and the first determined state ofthe door of the second vehicle, the first vehicle may be caused toinitiate a second action. In yet another example, in response to thedetermination that the second vehicle is in a lane and/or a planned pathof the first vehicle and a first determined state of the door of thesecond vehicle, the first vehicle may be caused to initiate a firstaction; and in response to the determination that the second vehicle isnot in a lane and/or is not in a planned path of the first vehicle andthe first determined state of the door of the second vehicle, causingthe first vehicle to initiate the first action may be withheld and/orforwent.

In some embodiments, systems, methods and computer readable media forcontrolling vehicles based on users of other vehicles are provided. Onechallenge of autonomous driving is to determine when a standing vehiclehas stopped for a short period of time and therefore the autonomousvehicle needs to wait for the standing vehicle to resume moving, andwhen the standing vehicle has stopped for a longer period of time andtherefore the autonomous vehicle needs to pass the standing vehicle. Insome cases, one indication of whether a vehicle has stopped for a longerperiod of time may include the state of the user. For example,identifying that the user is embarking and/or disembarking the vehiclemay indicate that the vehicle has stopped for a longer period of time.The provided systems, methods and computer readable media forcontrolling vehicles may detect a vehicle, identify a state of a user ofthe vehicle, and control a response of an autonomous vehicle to thedetected vehicle based on the identified state of the user.

FIG. 14 illustrates an example of a method 1400 for controlling vehiclesbased on users of other vehicles. In this example, method 1400 maycomprise: obtaining images captured from an environment of a firstvehicle (Step 710); analyzing the images to detect a second vehicle(Step 720); analyzing the images to determine a state of a userassociated with the second vehicle (Step 1430); in response to a firstdetermined state of the user, causing the first vehicle to initiate anaction responding to the second vehicle (Step 1440), and in response toa second determined state of the user, forgoing causing the firstvehicle to initiate the action. Some non-limiting examples of such userof the second vehicle may include a driver of the second vehicle, apassenger of the second vehicle, and so forth. In one example, thesecond vehicle detected by step 720 may be a garbage truck, and the usermay be a waste collector. In some implementations, method 1400 maycomprise one or more additional steps, while some of the steps listedabove may be modified or excluded. In some implementations, one or moresteps illustrated in FIG. 14 may be executed in a different order and/orone or more groups of steps may be executed simultaneously and/or aplurality of steps may be combined into single step and/or a single stepmay be broken down to a plurality of steps.

In some embodiments, one or more images captured using one or more imagesensors from an environment of a first vehicle may be obtained, forexample as described above. Further, in some examples, the one or moreimages may be analyzed to detect a second vehicle, for example asdescribed above. Further, in some examples, the one or more images maybe analyzed to determine a state of a user of the second vehicle.Further, in some examples, in response to a first determined state ofthe user of the second vehicle, the first vehicle may be caused toinitiate an action responding to the second vehicle.

In some embodiments, step 1430 may comprise analyzing the one or moreimages obtained by step 710 to determine a state of a user of the secondvehicle detected by step 720. Some non-limiting examples of such stateof the user of the second vehicle may include embarking the secondvehicle, disembarking the second vehicle, driving the second vehicle,sitting in the second vehicle, being in the second vehicle, and soforth. For example, a machine learning model may be trained usingtraining examples to determine states of users of vehicles from imagesand/or videos, and step 1430 may use the trained machine learning modelto analyze the one or more images obtained by step 710 and determine thestate of the user of the second vehicle. An example of such trainingexample may include an image and/or a video depicting a user of avehicle, together with a label indicating the state of the depicteduser. In another example, an artificial neural network (such as a deepneural networks, convolutional neural networks, etc.) may be configuredto determine states of users of vehicles from images and/or videos, andstep 1430 may use the artificial neural network to analyze the one ormore images obtained by step 710 and determine the state of the user ofthe second vehicle. In yet another example, step 1430 may use imageclassification algorithms to analyze the one or more images obtained bystep 710 and determine the state of the user of the second vehicle. Someother non-limiting examples of steps that may be used by step 1430 fordetermining the state of the user of the second vehicle are describedbelow.

In some examples, an orientation of at least part of the user of thesecond vehicle (in relation to at least one other part of the secondvehicle, in relation to the ground, in relation to the horizon, inrelation to an object in the environment, in relation to the at leastpart of the first vehicle, in relation to the image sensor, etc.) may bedetermined, and the determined state of the user of the second vehiclemay be based on the determined orientation of the at least part of theuser of the second vehicle. For example, a machine learning model may betrained using training examples to determine orientation of parts ofusers by analyzing images and/or videos, and the trained machinelearning model may be used to determine the orientation of the at leastpart of the user of the second vehicle from the one or more images. Inanother example, an artificial neural network (such as a deep neuralnetworks, convolutional neural networks, etc.) may be configured todetermine orientation of parts of users by analyzing images and/orvideos, and the configured artificial neural network may be used todetermine the orientation of the at least part of the user of the secondvehicle from the one or more images. In yet another example, informationabout the orientation of the at least part of the user of the secondvehicle may be received from the second vehicle (for example, using apoint to point communication protocol, through a communication networkusing a communication device, through a centralized server, and soforth).

In some examples, a distance of at least part of the user of the secondvehicle (from at least part of the second vehicle, from the ground, froman object in the environment, from at least part of the first vehicle,from the image sensor, etc.) may be determined, and the determined stateof the user of the second vehicle may be based on the determineddistance of the at least part of the user of the second vehicle. Forexample, a machine learning model may be trained using training examplesto determine distance of parts of users by analyzing images and/orvideos, and the trained machine learning model may be used to determinethe distance of the at least part of the user of the second vehicle fromthe one or more images. In another example, an artificial neural network(such as a deep neural networks, convolutional neural networks, etc.)may be configured to determine distance of parts of users by analyzingimages and/or videos, and the configured artificial neural network maybe used to determine the distance of the at least part of the user ofthe second vehicle from the one or more images. In yet another example,information about the distance of the at least part of the user of thesecond vehicle may be received from the second vehicle (for example,using a point to point communication protocol, through a communicationnetwork using a communication device, through a centralized server, andso forth).

In some examples, a motion of at least part of the user of the secondvehicle (in relation to at least one other part of the second vehicle,in relation to the ground, in relation to the horizon, in relation to anobject in the environment, in relation to the at least part of the firstvehicle, in relation to the image sensor, etc.) may be determined, andthe determined state of the user of the second vehicle may be based onthe determined motion of the at least part of the user of the secondvehicle. For example, a machine learning model may be trained usingtraining examples to determine motion of parts of users by analyzingimages and/or videos, and the trained machine learning model may be usedto determine the motion of the at least part of the user of the secondvehicle from the one or more images. In another example, an artificialneural network (such as a deep neural networks, convolutional neuralnetworks, etc.) may be configured to determine motion of parts of usersby analyzing images and/or videos, and the configured artificial neuralnetwork may be used to determine the motion of the at least part of theuser of the second vehicle from the one or more images. In yet anotherexample, information about the motion of the at least part of the userof the second vehicle may be received from the second vehicle (forexample, using a point to point communication protocol, through acommunication network using a communication device, through acentralized server, and so forth).

In some embodiments, step 1440 may comprise, for example in response toa first determined state of the user of the second vehicle detected bystep 720 (e.g., as determined by step 1430), causing the first vehicleto initiate an action responding to the second vehicle detected by step720. Some non-limiting examples of such action may include signaling,stopping, changing a speed of the first vehicle, changing a motiondirection of the first vehicle, passing the second vehicle, forgoingpassing the second vehicle, passing the user, keeping a minimal distanceof at least a selected length from the second vehicle (for example, theselected length may be less than 10 feet, may be at least 10 feet, maybe at least 20 feet, etc.), keeping a minimal distance of at least aselected length from the user (for example, the selected length may beless than 10 feet, may be at least 10 feet, may be at least 20 feet,etc.), turning, performing a U-turn, driving in reverse, generating anaudible warning, and so forth. For example, step 1440 may transmit asignal to an external device (such as a device controlling the firstvehicle, a device navigating the first vehicle, the first vehicle, asystem within the first vehicle, etc.), and the signal may be configuredto cause the external device to cause the first vehicle to initiate theaction responding to the second vehicle detected by step 720. In anotherexample, step 1440 may provide information related to the second vehicledetected by step 720 (such as position, motion, acceleration, type,dimensions, etc.) to such external device, and the information may beconfigured to cause the external device to cause the first vehicle toinitiate the action responding to the second vehicle detected by step720. In one example, in response a second state of the user of thesecond vehicle detected by step 720 (e.g., as determined by step 1430),step 1440 may cause the first vehicle to initiate a second action, thesecond action may differ from the action.

In some examples, step 1440 may select the action based on thedetermined state of the user of the second vehicle detected by step 720(e.g., as determined by step 1430). For example, in response to a firststate of the user of the second vehicle detected by step 720 (e.g., asdetermined by step 1430), step 1440 may select a first action, and inresponse to a second state of the user of the second vehicle detected bystep 720 (e.g., as determined by step 1430), step 1440 may select asecond action (where the second action may differ from the first action,and the second state of the user of the second vehicle may differ fromthe first state of the user of the second vehicle). In some examples, inresponse to a first state of the user of the second vehicle detected bystep 720 (e.g., as determined by step 1430), step 1440 may select afirst action, and in response to a second state of the user of thesecond vehicle detected by step 720 (e.g., as determined by step 1430),step 1440 may withhold and/or forgo the first action (for example, step1440 may withhold and/or forgo causing the first vehicle to initiate theaction).

In some examples, a motion of the second vehicle may be determined, forexample as described above. Further, in some examples, in response tothe determined motion of the second vehicle and the first determinedstate of the user of the second vehicle, the first vehicle may be causedto initiate the action (for example as described above). For example, inresponse to a first determined motion of the second vehicle and a firstdetermined state of the user of the second vehicle, the first vehiclemay be caused to initiate a first action; in response to a seconddetermined motion of the second vehicle and the first determined stateof the user of the second vehicle, the first vehicle may be caused toinitiate a second action; and in response to the first determined motionof the second vehicle and a second determined state of the user of thesecond vehicle, the first vehicle may be caused to initiate a thirdaction. In another example, in response to a first determined motion ofthe second vehicle and a first determined state of the user of thesecond vehicle, the first vehicle may be caused to initiate a firstaction; in response to a second determined motion of the second vehicleand the first determined state of the user of the second vehicle and/orin response to the first determined motion of the second vehicle and asecond determined state of the user of the second vehicle, causing thefirst vehicle to initiate the first action may be withheld and/orforwent.

In some examples, it may be determining that the second vehicle issignaling, for example as described above. Further, in some examples, atype of the signaling of the second vehicle may be determined, forexample as described above. Some non-limiting examples of such type ofsignaling may include signaling using light, signaling using sound,signaling left, signaling right, signaling break, signaling hazardwarning, signaling reversing, signaling warning, and so forth. Further,in some examples, in response to the determination that the secondvehicle is signaling and/or the determined type of the signaling and/orthe first determined state of the user of the second vehicle, the firstvehicle may be caused to initiate the action (for example as describedabove). For example, in response to a first determined type of thesignaling and a first determined state of the user of the secondvehicle, the first vehicle may be caused to initiate a first action; inresponse to a second determined type of the signaling and the firstdetermined state of the user of the second vehicle, the first vehiclemay be caused to initiate a second action; and in response to the firstdetermined type of the signaling and a second determined state of theuser of the second vehicle, the first vehicle may be caused to initiatea third action. In another example, in response to a first determinedtype of the signaling and a first determined state of the user of thesecond vehicle, the first vehicle may be caused to initiate a firstaction; in response to a second determined type of the signaling and thefirst determined state of the user of the second vehicle and/or inresponse to the first determined type of the signaling and a seconddetermined state of the user of the second vehicle, causing the firstvehicle to initiate the first action may be withheld and/or forwent. Inan additional example, in response to the determination that the secondvehicle is signaling and a first determined state of the user of thesecond vehicle, the first vehicle may be caused to initiate a firstaction; and in response to the determination that the second vehicle isnot signaling and the first determined state of the user of the secondvehicle, the first vehicle may be caused to initiate a second action. Inyet another example, in response to the determination that the secondvehicle is signaling and a first determined state of the user of thesecond vehicle, the first vehicle may be caused to initiate a firstaction; and in response to the determination that the second vehicle isnot signaling and the first determined state of the user of the secondvehicle, causing the first vehicle to initiate the first action may bewithheld and/or forwent.

In some examples, a position of the second vehicle may be determined,for example as described above. For example, the determined position maybe in relation to the ground, in relation to a map, in relation to aroad, in relation to a lane, in relation to an object in theenvironment, in relation to the at least part of the first vehicle, inrelation to the image sensor, and so forth. Further, in some examples,it may be determining that the second vehicle is in a lane of the firstvehicle, for example as described above. Further, in some examples, itmay be determining that the second vehicle is in a planned path of thefirst vehicle, for example as described above. Further, in someexamples, in response to the determined position of the second vehicleand the first determined state of the user of the second vehicle, thefirst vehicle may be caused to initiate the action (for example asdescribed above). For example, in response to a first determinedposition of the second vehicle and a first determined state of the userof the second vehicle, the first vehicle may be caused to initiate afirst action; in response to a second determined position of the secondvehicle and the first determined state of the user of the secondvehicle, the first vehicle may be caused to initiate a second action;and in response to the first determined position of the second vehicleand a second determined state of the user of the second vehicle, thefirst vehicle may be caused to initiate a third action. In anotherexample, in response to a first determined position of the secondvehicle and a first determined state of the user of the second vehicle,the first vehicle may be caused to initiate a first action; in responseto a second determined position of the second vehicle and the firstdetermined state of the user of the second vehicle and/or in response tothe first determined position of the second vehicle and a seconddetermined state of the user of the second vehicle, causing the firstvehicle to initiate the first action may be withheld and/or forwent. Inan additional example, in response to the determination that the secondvehicle is in a lane and/or a planned path of the first vehicle and afirst determined state of the user of the second vehicle, the firstvehicle may be caused to initiate a first action; and in response to thedetermination that the second vehicle is not in a lane and/or is not ina planned path of the first vehicle and the first determined state ofthe user of the second vehicle, the first vehicle may be caused toinitiate a second action. In yet another example, in response to thedetermination that the second vehicle is in a lane and/or a planned pathof the first vehicle and a first determined state of the user of thesecond vehicle, the first vehicle may be caused to initiate a firstaction; and in response to the determination that the second vehicle isnot in a lane and/or is not in a planned path of the first vehicle andthe first determined state of the user of the second vehicle, causingthe first vehicle to initiate the first action may be withheld and/orforwent.

In some embodiments, the one or more images obtained by step 710 may beanalyzed to detect a person in a vicinity of the second vehicle detectedby step 720, for example as described below. Further, in some examples,the one or more images obtained by step 710 may be analyzed to determinethat the person is speaking with the user, for example as describedbelow. Further, in some examples, step 1440 may cause the first vehicleto initiate an action responding to the second vehicle detected by step720, for example in response to the determination that the person isspeaking with the user.

In some examples, the one or more images obtained by step 710 may beanalyzed to detect a person in a vicinity of the second vehicle detectedby step 720. For example, a machine learning model may be trained usingtraining examples to detect persons in vicinity of vehicles by analyzingimages and/or videos, and the trained machine learning model may be usedto analyze the one or more images obtained by step 710 and detectpersons in the vicinity of the second vehicle detected by step 720. Anexample of such training example may include an image and/or a videodepicting a vehicle, together with a label indicating whether a personis present in the vicinity of the depicted vehicle. In another example,an artificial neural network (such as a deep neural networks,convolutional neural networks, etc.) may be configured to detect personsin vicinity of vehicles by analyzing images and/or videos, and theartificial neural network may be used to analyze the one or more imagesobtained by step 710 and detect persons in the vicinity of the secondvehicle detected by step 720. In yet another example, person detectionalgorithms may be used to detect people in the one or more imagesobtained by step 710, the location of the second vehicle detected bystep 720 may be compared with the locations of the detected people todetermine whether any of the detected people are in the vicinity of thesecond vehicle detected by step 720.

In some examples, the one or more images obtained by step 710 may beanalyzed to determine whether a person in the vicinity of the secondvehicle detected by step 720 is speaking with a user of the secondvehicle. For example, the one or more images obtained by step 710 may beanalyzed to determine whether the person detected in the vicinity of thesecond vehicle detected by step 720 is speaking with a user of thesecond vehicle. For example, a machine learning model may be trainedusing training examples to determine whether persons are talking withusers of vehicles by analyzing images and/or videos, and the trainedmachine learning model may be used to analyze the one or more imagesobtained by step 710 and determine whether a person in the vicinity ofthe second vehicle detected by step 720 is speaking with a user of thesecond vehicle. An example of such training example may include an imageand/or a video depicting a vehicle and a person in the vicinity of thevehicle, together with a label indicating whether the depicting personis speaking with a user of the depicted vehicle. In another example, anartificial neural network (such as a deep neural networks, convolutionalneural networks, etc.) may be configured to determine whether personsare talking with users of vehicles by analyzing images and/or videos,and the artificial neural network may be used to analyze the one or moreimages obtained by step 710 and determine whether a person in thevicinity of the second vehicle detected by step 720 is speaking with auser of the second vehicle.

In some examples, for example in response to the determination that theperson is speaking with the user, step 1440 may cause the first vehicleto initiate an action responding to the second vehicle, for example asdescribed above. For example, in response to the determination that theperson is speaking with the user, step 1440 may cause the first vehicleto initiate a first action, and in response to the determination thatthe person is not speaking with the user, step 1440 may cause the firstvehicle to initiate a second action. For example, in response to thedetermination that the person is speaking with the user, step 1440 maycause the first vehicle to initiate a first action, and in response tothe determination that the person is not speaking with the user, step1440 may withhold and/or forgo causing the first vehicle to initiate thefirst action may be withheld and/or forwent.

In some embodiments, systems, methods and computer readable media forcontrolling vehicles based on loading and/or unloading events of othervehicles are provided. One challenge of autonomous driving is todetermine when a standing vehicle has stopped for a short period of timeand therefore the autonomous vehicle needs to wait for the standingvehicle to resume moving, and when the standing vehicle has stopped fora longer period of time and therefore the autonomous vehicle needs topass the standing vehicle. In some cases, identifying cargo loading orunloading event corresponding to the vehicle may indicate that thevehicle has stopped for a longer period of time. The provided systems,methods and computer readable media for controlling vehicles detect avehicle, identify a cargo loading or unloading event, and control aresponse of an autonomous vehicle to the detected vehicle based on theidentified event.

FIG. 15 illustrates an example of a method 1500 for controlling vehiclesbased on loading and/or unloading events. In this example, method 1500may comprise: obtaining images captured from an environment of a firstvehicle (Step 710); analyzing the images to detect a second vehicle(Step 720); analyzing the images to determine at least one of a cargoloading event associated with the second vehicle and a cargo unloadingevent associated with the second vehicle (Step 1530); in response to thedetermined at least one of cargo loading event associated with thesecond vehicle and cargo unloading event associated with the secondvehicle, causing the first vehicle to initiate an action responding tothe second vehicle (Step 1540), and in response to no determined cargoloading event associated with the second vehicle and no determined cargounloading event associated with the second vehicle, forgoing causing thefirst vehicle to initiate the action. In some implementations, method1500 may comprise one or more additional steps, while some of the stepslisted above may be modified or excluded. In some implementations, oneor more steps illustrated in FIG. 15 may be executed in a differentorder and/or one or more groups of steps may be executed simultaneouslyand/or a plurality of steps may be combined into single step and/or asingle step may be broken down to a plurality of steps.

In some embodiments, one or more images captured using one or more imagesensors from an environment of a first vehicle may be obtained, forexample as described above. Further, in some examples, the one or moreimages may be analyzed to detect a second vehicle, for example asdescribed above. Further, in some examples, at least one of a cargoloading event associated with the second vehicle and a cargo unloadingevent associated with the second vehicle may be determined. Further, insome examples, for example in response to the determined at least one ofcargo loading event associated with the second vehicle and cargounloading event associated with the second vehicle, the first vehiclemay be caused to initiate an action responding to the second vehicle,for example as described above.

In some examples, step 1530 may comprise analyzing the one or moreimages obtained by step 710 to determine a cargo loading eventassociated with the second vehicle detected by step 720 and/or a cargounloading event associated with the second vehicle detected by step 720.For example, a machine learning model may be trained using trainingexamples to determine cargo loading events and/or cargo unloading eventsand/or properties of the events by analyzing images and/or videos, andstep 1530 may use the trained machine learning model to analyze the oneor more images obtained by step 710 and determine the cargo loadingevent associated with the second vehicle detected by step 720 and/or thecargo unloading event associated with the second vehicle detected bystep 720 and/or properties of the event. An example of such trainingexample may include an image and/or a video depicting a vehicle,together with a label indicating an occurrence of cargo loading eventassociated with the depicted vehicle and/or an occurrence of cargounloading event associated with the depicted vehicle and/or propertiesof the event. In another example, an artificial neural network (such asa deep neural networks, convolutional neural networks, etc.) may beconfigured to determine cargo loading events and/or cargo unloadingevents and/or properties of the events by analyzing images and/orvideos, and step 1530 may use the artificial neural network to analyzethe one or more images obtained by step 710 and determine the cargoloading event associated with the second vehicle detected by step 720and/or the cargo unloading event associated with the second detected bystep 720 vehicle and/or properties of the event. In yet another example,step 1530 may use event detection algorithms and/or event recognitionalgorithms to analyze the one or more images obtained by step 710 anddetect the cargo loading event associated with the second vehicledetected by step 720 and/or the cargo unloading event associated withthe second vehicle detected by step 720 and/or properties of the event,for example by detecting cargo loading events and/or cargo unloadingevents in the vicinity of the second vehicle. Some non-limiting examplesof such properties of cargo loading events and/or cargo unloading eventsmay include side of the second vehicle associated with the event, atiming of the event, type of cargo, and so forth.

In some examples, step 1530 may comprise receiving an indication of thecargo loading event associated with the second vehicle detected by step720 and/or of the cargo unloading event associated with the secondvehicle detected by step 720 and/or information about the event (such asside of the second vehicle associated with the event, a timing of theevent, type of cargo, and so forth), for example from the second vehicledetected by step 720 (for example, using a point to point communicationprotocol, through a communication network using a communication device,through a centralized server, and so forth), from an external device(such as an external device associated with the second vehicle detectedby step 720). Further, in some examples, step 1530 may determine thecargo loading event associated with the second vehicle detected by step720 and/or the cargo unloading event associated with the second vehicledetected by step 720 based on the received indication and/or based onthe received information.

In some embodiments, step 1540 may comprise, for example in response tothe determined cargo loading event associated with the second vehicledetected by step 720 (e.g., as determined by step 1530) and/or thedetermined cargo unloading event associated with the second vehicledetected by step 720 (e.g., as determined by step 1530), causing thefirst vehicle to initiate an action responding to the second vehicledetected by step 720. Some non-limiting examples of such action mayinclude signaling, stopping, changing a speed of the first vehicle,changing a motion direction of the first vehicle, passing the secondvehicle, forgoing passing the second vehicle, passing the cargo, passinga person related to the event, keeping a minimal distance of at least aselected length from the second vehicle (for example, the selectedlength may be less than 10 feet, may be at least 10 feet, may be atleast 20 feet, etc.), keeping a minimal distance of at least a selectedlength from the cargo (for example, the selected length may be less than10 feet, may be at least 10 feet, may be at least 20 feet, etc.),keeping a minimal distance of at least a selected length from a personrelated to the event (for example, the selected length may be less than10 feet, may be at least 10 feet, may be at least 20 feet, etc.),turning, performing a U-turn, driving in reverse, generating an audiblewarning, and so forth. For example, in response to the determined cargoloading event associated with the second vehicle detected by step 720(e.g., as determined by step 1530), step 1540 may cause the firstvehicle to initiate a first action, and in response to the determinedcargo unloading event associated with the second vehicle detected bystep 720 (e.g., as determined by step 1530), step 1540 may the firstvehicle to initiate a second vehicle, where in one example the firstaction and the second action may be the same action, and in anotherexample the first action may differ from the second action. In someexamples, in response to the determined cargo loading event associatedwith the second vehicle detected by step 720 (e.g., as determined bystep 1530) and/or the determined cargo unloading event associated withthe second vehicle detected by step 720 (e.g., as determined by step1530), step 1540 may cause the first vehicle to initiate a first action,and in response to a determination that no such event occurs, step 1540may cause the first vehicle to initiate a second action, where thesecond action may differ from the first action. In some examples, inresponse to the determined cargo loading event associated with thesecond vehicle detected by step 720 (e.g., as determined by step 1530)and/or the determined cargo unloading event associated with the secondvehicle detected by step 720 (e.g., as determined by step 1530), step1540 may cause the first vehicle to initiate a first action, and inresponse to a determination that no such event occurs, step 1540 maywithhold and/or forgo causing the first vehicle to initiate the firstaction.

In some embodiments, step 1540 may select the action based on thedetermined side of the second vehicle detected by step 720 associatedwith the event determined by step 1530. Some non-limiting examples ofsuch sides of the second vehicle detected by step 720 may include left,right, back, front, and so forth. For example, in response to a firstdetermined side, step 1540 may cause the first vehicle to initiate afirst action, and in response to a second determined side, step 1540 maycause the first vehicle to initiate a second action, where the secondaction may differ from the first action. In another example, in responseto a first determined side, step 1540 may cause the first vehicle toinitiate a first action, and in response to a second determined side,step 1540 may withhold and/or forgo causing the first vehicle toinitiate the first action.

In some embodiments, the one or more images obtained by step 710 may beanalyzed to detect a person associated with the determined (e.g., bystep 1530) cargo loading event associated with the second vehicle and/orwith the determined (e.g., by step 1530) cargo unloading eventassociated with the second vehicle. Further, in some examples, the oneor more images obtained by step 710 may be analyzed to determine aposition of the detected person (for example, with respect to the firstvehicle, with respect to the second vehicle detected by step 720, in theone or more images, an absolute position, and so forth). Further, insome examples, step 1540 may select the action based on the determinedposition of the detected person. For example, in response to a firstdetermined position, step 1540 may select a first action, and inresponse to a second determined position, step 1540 may select a secondaction, the second action may differ from the first action. In anotherexample, in response to a first determined position, step 1540 mayselect a first action, and in response to a second determined position,step 1540 may withhold and/or forgo the first action.

In some embodiments, the one or more images obtained by step 710 may beanalyzed to detect a person associated with the determined (e.g., bystep 1530) cargo loading event associated with the second vehicle and/orwith the determined (e.g., by step 1530) cargo unloading eventassociated with the second vehicle. Further, in some examples, the oneor more images obtained by step 710 may be analyzed to determine anorientation of the detected person (for example, with respect to thefirst vehicle, with respect to the second vehicle detected by step 720,with respect to the ground, with respect to the image sensor used tocapture the one or more images obtained by step 710, and so forth).Further, in some examples, step 1540 may select the action based on thedetermined orientation of the detected person. For example, in responseto a first determined orientation, step 1540 may select a first action,and in response to a second determined orientation, step 1540 may selecta second action, the second action may differ from the first action. Inanother example, in response to a first determined orientation, step1540 may select a first action, and in response to a second determinedorientation, step 1540 may withhold and/or forgo the first action.

In some examples, the one or more images obtained by step 710 may beanalyzed to detect a person associated with the determined (e.g., bystep 1530) cargo loading event associated with the second vehicle and/orwith the determined (e.g., by step 1530) cargo unloading eventassociated with the second vehicle. Further, in some examples, the oneor more images obtained by step 710 may be analyzed to determine amotion the detected person (for example, with respect to the firstvehicle, with respect to the second vehicle detected by step 720, withrespect to the ground, with respect to the image sensor used to capturethe one or more images obtained by step 710, and so forth). Further, insome examples, step 1540 may select the action based on the determinedmotion of the detected person. For example, in response to a firstdetermined motion, step 1540 may select a first action, and in responseto a second determined motion, step 1540 may select a second action, thesecond action may differ from the first action. In another example, inresponse to a first determined motion, step 1540 may select a firstaction, and in response to a second determined motion, step 1540 maywithhold and/or forgo the first action.

In some examples, the one or more images obtained by step 710 may beanalyzed to detect a person associated with the determined cargo loadingevent associated with the second vehicle and/or with the determinedcargo unloading event associated with the second vehicle. For example, amachine learning model may be trained using training examples to detectpersons involved in cargo loading events and/or in cargo unloadingevents by analyzing images and/or videos, and the trained machinelearning model may be used to analyze the one or more images obtained bystep 710 and detect persons involved in the cargo loading event and/orcargo unloading event associated with the second vehicle detected bystep 720. An example of such training example may include an imageand/or a video depicting a cargo loading event and/or a cargo unloadingevent associated with a vehicle, together with a label indicating aperson depicted in the image and/or in the video and involved in thedepicted event. In another example, an artificial neural network (suchas a deep neural networks, convolutional neural networks, etc.) may beconfigured to detect persons involved in cargo loading events and/or incargo unloading events by analyzing images and/or videos, and theartificial neural network may be used to analyze the one or more imagesobtained by step 710 and detect persons involved in the cargo loadingevent and/or cargo unloading event associated with the second vehicledetected by step 720.

In some examples, the one or more images obtained by step 710 may beanalyzed to determine a position of the detected person (for example,with respect to the first vehicle, with respect to the second vehicledetected by step 720, in the one or more images, an absolute position,and so forth). For example, the position of the person in the one ormore images obtained by step 710 may be determined using persondetection algorithms. In another example, the detection methodsdescribed above may be further trained and/or configured to determine aposition of the detected person.

In some examples, the one or more images obtained by step 710 may beanalyzed to determine an orientation of the detected person (forexample, with respect to the first vehicle, with respect to the secondvehicle detected by step 720, with respect to the ground, with respectto the image sensor used to capture the one or more images obtained bystep 710, and so forth). For example, the detection methods describedabove may be further trained and/or configured to determine anorientation of the detected person.

In some examples, the one or more images obtained by step 710 may beanalyzed to determine a motion of the detected person (for example, withrespect to the first vehicle, with respect to the second vehicledetected by step 720, in the one or more images, an absolute motion, andso forth). For example, the motion of the person in the one or moreimages obtained by step 710 may be determined using person detectionalgorithms. In another example, the detection methods described abovemay be further trained and/or configured to determine a motion of thedetected person.

In some examples, the action may be selected based on the determinedmotion of the detected person in relation to the second vehicle. Forexample, in response to a first determined motion, a first action may beselected, and in response to a second determined motion, a second actionmay be selected. In another example, in response to a first determinedmotion, a first action may be selected, and in response to a seconddetermined motion, the first action may be withheld and/or forwent.

In some examples, a motion of the second vehicle may be determined, forexample as described above. Further, in some examples, in response tothe determined motion of the second vehicle and/or the determined cargoloading event associated with the second vehicle and/or the determinedcargo unloading event associated with the second vehicle and/or based onthe determined properties of the event, the first vehicle may be causedto initiate the action (for example as described above). For example, inresponse to a first determined motion of the second vehicle and a firstdetermined event, the first vehicle may be caused to initiate a firstaction; in response to a second determined motion of the second vehicleand the first determined event, the first vehicle may be caused toinitiate a second action; and in response to the first determined motionof the second vehicle and a second determined event (for example, acargo loading event in contrast to a cargo unloading event, an eventassociated with a different side of the second vehicle, etc.), the firstvehicle may be caused to initiate a third action. In another example, inresponse to a first determined motion of the second vehicle and a firstdetermined event, the first vehicle may be caused to initiate a firstaction; in response to a second determined motion of the second vehicleand the first determined event and/or in response to the firstdetermined motion of the second vehicle and a second determined event(for example, a cargo loading event in contrast to a cargo unloadingevent, an event associated with a different side of the second vehicle,etc.), causing the first vehicle to initiate the first action may bewithheld and/or forwent.

In some examples, it may be determining that the second vehicle issignaling, for example as described above. Further, in some examples, atype of the signaling of the second vehicle may be determined, forexample as described above. Some non-limiting examples of such type ofsignaling may include signaling using light, signaling using sound,signaling left, signaling right, signaling break, signaling hazardwarning, signaling reversing, signaling warning, and so forth. Further,in some examples, in response to the determination that the secondvehicle is signaling and/or the determined type of the signaling and/orthe determined cargo loading event associated with the second vehicleand/or the determined cargo unloading event associated with the secondvehicle and/or based on the determined properties of the event, thefirst vehicle may be caused to initiate the action (for example asdescribed above). For example, in response to a first determined type ofthe signaling and a first determined event, the first vehicle may becaused to initiate a first action; in response to a second determinedtype of the signaling and the first determined event, the first vehiclemay be caused to initiate a second action; and in response to the firstdetermined type of the signaling and a second determined event (forexample, a cargo loading event in contrast to a cargo unloading event,an event associated with a different side of the second vehicle, etc.),the first vehicle may be caused to initiate a third action. In anotherexample, in response to a first determined type of the signaling and afirst determined event, the first vehicle may be caused to initiate afirst action; in response to a second determined type of the signalingand the first determined event and/or in response to the firstdetermined type of the signaling and a second determined event (forexample, a cargo loading event in contrast to a cargo unloading event,an event associated with a different side of the second vehicle, etc.),causing the first vehicle to initiate the first action may be withheldand/or forwent. In an additional example, in response to thedetermination that the second vehicle is signaling and a firstdetermined event, the first vehicle may be caused to initiate a firstaction; and in response to the determination that the second vehicle isnot signaling and the first determined event, the first vehicle may becaused to initiate a second action. In yet another example, in responseto the determination that the second vehicle is signaling and a firstdetermined event, the first vehicle may be caused to initiate a firstaction; and in response to the determination that the second vehicle isnot signaling and the first determined event, causing the first vehicleto initiate the first action may be withheld and/or forwent.

In some examples, a position of the second vehicle may be determined,for example as described above. For example, the determined position maybe in relation to the ground, in relation to a map, in relation to aroad, in relation to a lane, in relation to an object in theenvironment, in relation to the at least part of the first vehicle, inrelation to the image sensor, and so forth. Further, in some examples,it may be determining that the second vehicle is in a lane of the firstvehicle, for example as described above. Further, in some examples, itmay be determining that the second vehicle is in a planned path of thefirst vehicle, for example as described above. Further, in someexamples, in response to the determined position of the second vehicleand/or the determined cargo loading event associated with the secondvehicle and/or the determined cargo unloading event associated with thesecond vehicle and/or based on the determined properties of the event,the first vehicle may be caused to initiate the action (for example asdescribed above). For example, in response to a first determinedposition of the second vehicle and a first determined event, the firstvehicle may be caused to initiate a first action; in response to asecond determined position of the second vehicle and the firstdetermined event, the first vehicle may be caused to initiate a secondaction; and in response to the first determined position of the secondvehicle and a second determined event (for example, a cargo loadingevent in contrast to a cargo unloading event, an event associated with adifferent side of the second vehicle, etc.), the first vehicle may becaused to initiate a third action. In another example, in response to afirst determined position of the second vehicle and a first determinedevent, the first vehicle may be caused to initiate a first action; inresponse to a second determined position of the second vehicle and thefirst determined event and/or in response to the first determinedposition of the second vehicle and a second determined event (forexample, a cargo loading event in contrast to a cargo unloading event,an event associated with a different side of the second vehicle, etc.),causing the first vehicle to initiate the first action may be withheldand/or forwent. In an additional example, in response to thedetermination that the second vehicle is in a lane and/or a planned pathof the first vehicle and a first determined event, the first vehicle maybe caused to initiate a first action; and in response to thedetermination that the second vehicle is not in a lane and/or is not ina planned path of the first vehicle and the first determined event, thefirst vehicle may be caused to initiate a second action. In yet anotherexample, in response to the determination that the second vehicle is ina lane and/or a planned path of the first vehicle and a first determinedevent, the first vehicle may be caused to initiate a first action; andin response to the determination that the second vehicle is not in alane and/or is not in a planned path of the first vehicle and the firstdetermined event, causing the first vehicle to initiate the first actionmay be withheld and/or forwent.

In some embodiments, step 1530 may comprise determining a cargo loadingevent associated with the second vehicle detected by step 720. Further,in some examples, for example in response to the determined cargoloading event associated with the second vehicle, step 1540 may causethe first vehicle to initiate an action responding to the secondvehicle, for example as described above. Some non-limiting examples ofsuch action may include signaling, stopping, changing a speed of thefirst vehicle, changing a motion direction of the first vehicle, passingthe second vehicle, forgoing passing the second vehicle, turning,performing a U-turn, driving in reverse, generating an audible warning,and so forth.

In some embodiments, step 1530 may comprise determining a cargounloading event associated with the second vehicle detected by step 720.Further, in some examples, for example in response to the determinedcargo unloading event associated with the second vehicle, step 1540 maycause the first vehicle to initiate an action responding to the secondvehicle, for example as described above. Some non-limiting examples ofsuch action may include signaling, stopping, changing a speed of thefirst vehicle, changing a motion direction of the first vehicle, passingthe second vehicle, forgoing passing the second vehicle, turning,performing a U-turn, driving in reverse, generating an audible warning,and so forth.

In some embodiments, systems, methods and computer readable media forcontrolling vehicles in response to street cleaning vehicles areprovided. One challenge of autonomous driving is to determine when avehicle is moving slowly for a short period of time and therefore theautonomous vehicle needs to wait for the slow vehicle to resume normalspeed, and when the vehicle is moving slowly for a longer period of timeand the autonomous vehicle needs to pass the slow moving vehicle. Insome cases, identifying that the vehicle is a street cleaning vehiclemay indicate that the vehicle is moving slowly for a longer period oftime. The provided systems, methods and computer readable media forcontrolling vehicles may detect a vehicle, identify whether the vehicleis a street cleaning vehicle, and control a response of an autonomousvehicle to the detected vehicle based on whether the vehicle is a streetcleaning vehicle.

FIG. 16 illustrates an example of a method 1600 for controlling vehiclesin response to street cleaning vehicles. In this example, method 1600may comprise: obtaining images captured from an environment of a firstvehicle (Step 710); analyzing the images to detect a second vehicle(Step 720); analyzing the images to determine that the second vehicle isa street cleaning vehicle (Step 1630); in response to the determinationthat the second vehicle is a street cleaning vehicle, causing the firstvehicle to initiate an action responding to the second vehicle (Step1640), and in response to a determination that the second vehicle is nota street cleaning vehicle, forgoing causing the first vehicle toinitiate the action. In some implementations, method 1600 may compriseone or more additional steps, while some of the steps listed above maybe modified or excluded. In some implementations, one or more stepsillustrated in FIG. 16 may be executed in a different order and/or oneor more groups of steps may be executed simultaneously and/or aplurality of steps may be combined into single step and/or a single stepmay be broken down to a plurality of steps.

In some embodiments, one or more images captured using one or more imagesensors from an environment of a first vehicle may be obtained, forexample as described above. Further, in some examples, the one or moreimages may be analyzed to detect a second vehicle, for example asdescribed above. Further, in some examples, the one or more images maybe analyzed to determine whether the second vehicle is a street cleaningvehicle. Further, in examples, for example in response to thedetermination that the second vehicle is a street cleaning vehicle, thefirst vehicle may be caused to initiate an action responding to thesecond vehicle, for example as described above.

In some embodiments, step 1630 may comprise analyzing the one or moreimages obtained by step 710 to determine whether the second vehicledetected by step 720 is a street cleaning vehicle. For example, amachine learning model may be trained using training examples todetermine whether vehicles are street cleaning vehicles by analyzingimages and/or videos, and step 1630 may use the trained machine learningmodel to analyze the one or more images obtained by step 710 anddetermine whether the second vehicle detected by step 720 is a streetcleaning vehicle. An example of such training example may include animage and/or a video depicting a vehicle, together with a labelindicating whether the depicted vehicle is a street cleaning vehicle. Inanother example, an artificial neural network (such as a deep neuralnetworks, convolutional neural networks, etc.) may be configured todetermine whether vehicles are street cleaning vehicles by analyzingimages and/or videos, and step 1630 may use the artificial neuralnetwork to analyze the one or more images obtained by step 710 anddetermine whether the second vehicle detected by step 720 is a streetcleaning vehicle. In yet another example, step 1630 may use imageclassification algorithms to analyze the one or more images obtained bystep 710 and determine whether the second vehicle detected by step 720is a street cleaning vehicle.

In some embodiments, step 1640 may comprise, for example in response toa determination that the second vehicle detected by step 720 is a streetcleaning vehicle (e.g., as determined by step 1630), causing the firstvehicle to initiate an action responding to the second vehicle detectedby step 720. Some non-limiting examples of such action may includesignaling, stopping, changing a speed of the first vehicle, changing amotion direction of the first vehicle, passing the second vehicle,forgoing passing the second vehicle, keeping a minimal distance of atleast a selected length from the second vehicle (for example, theselected length may be less than 100 feet, may be at least 100 feet, maybe at least 200 feet, etc.), turning, performing a U-turn, driving inreverse, generating an audible warning, and so forth. For example, step1640 may transmit a signal to an external device (such as a devicecontrolling the first vehicle, a device navigating the first vehicle,the first vehicle, a system within the first vehicle, etc.), and thesignal may be configured to cause the external device to cause the firstvehicle to initiate the action responding to the second vehicle detectedby step 720. In another example, step 1640 may provide informationrelated to the second vehicle detected by step 720 (such as position,motion, type, activity status, dimensions, etc.) to such externaldevice, and the information may be configured to cause the externaldevice to cause the first vehicle to initiate the action responding tothe second vehicle detected by step 720. In one example, in response toa determination that the second vehicle detected by step 720 is not astreet cleaning vehicle (e.g., as determined by step 1630), step 1640may cause the first vehicle to initiate a second action, the secondaction may differ from the action. In one example, in response to adetermination that the second vehicle detected by step 720 is a streetcleaning vehicle (e.g., as determined by step 1630), step 1640 may causethe first vehicle to initiate a first action, and in response to adetermination that the second vehicle detected by step 720 is not astreet cleaning vehicle (e.g., as determined by step 1630), step 1640may cause the first vehicle to initiate a second action, the secondaction may differ from the first action. In another example, in responseto a determination that the second vehicle detected by step 720 is astreet cleaning vehicle (e.g., as determined by step 1630), step 1640may cause the first vehicle to initiate a first action, and in responseto a determination that the second vehicle detected by step 720 is not astreet cleaning vehicle (e.g., as determined by step 1630), step 1640may withhold and/or forgo causing the first vehicle to initiate thefirst action.

In some embodiments, a state of the street cleaning vehicle may bedetermined. Some non-limiting examples of such state may includecleaning, not cleaning, and so forth. In some examples, the one or moreimages may be analyzed to determine a state of the street cleaningvehicle (such as ‘cleaning’, ‘not cleaning’, and so forth). For example,a machine learning model may be trained using training examples todetermine states of street cleaning vehicles by analyzing images and/orvideos, and the trained machine learning model may be used to analyzethe one or more images obtained by step 710 and determine the state ofthe street cleaning vehicle. An example of such training example mayinclude an image and/or a video depicting a street cleaning vehicle,together with a label indicating a state of the depicted street cleaningvehicle. In another example, an artificial neural network (such as adeep neural networks, convolutional neural networks, etc.) may beconfigured to determine states of street cleaning vehicles by analyzingimages and/or videos, and the artificial neural network may be used toanalyze the one or more images obtained by step 710 and determine thestate of the street cleaning vehicle. In yet another example, imageclassification algorithms may be used to analyze the one or more imagesobtained by step 710 and determine the state of the street cleaningvehicle.

In some examples, audio data captured using one or more audio sensorsfrom an environment of the first vehicle may be obtained and/or analyzedto determine a state of the street cleaning vehicle (such as ‘cleaning’,‘not cleaning’, and so forth). For example, a machine learning model maybe trained using training examples to determine states of streetcleaning vehicles by analyzing audio input, and the trained machinelearning model may be used to analyze the captured audio data anddetermine the state of the street cleaning vehicle. An example of suchtraining example may include an audio recording of a street cleaningvehicle, together with a label indicating a state of the recorded streetcleaning vehicle. In another example, an artificial neural network (suchas a recurrent neural networks, long short-term memory neural networks,etc.) may be configured to determine states of street cleaning vehiclesby analyzing audio input, and the artificial neural network may be usedto analyze the audio data and determine the state of the street cleaningvehicle. In yet another example, signal processing algorithms may beused to analyze the audio data and determine whether the street cleaningvehicle is cleaning or not.

In some examples, in response to a first determined state of the streetcleaning vehicle, step 1640 may cause the first vehicle to initiate afirst action responding to the street cleaning vehicle, and in responseto a second determined state of the street cleaning vehicle, step 1640may withhold and/or forgo causing the first vehicle to initiate thefirst action. In some examples, in response to a first determined stateof the street cleaning vehicle, step 1640 may cause the first vehicle toinitiate a first action responding to the street cleaning vehicle, andin response to a second determined state of the street cleaning vehicle,step 1640 may cause the first vehicle to initiate a second actionresponding to the street cleaning vehicle, where the second action maydiffer from the first action.

In some examples, a motion of the second vehicle may be determined, forexample as described above. Further, in some examples, in response tothe determined motion of the second vehicle and/or the determinationthat the second vehicle is a street cleaning vehicle and/or based on thedetermined state of the street cleaning vehicle, the first vehicle maybe caused to initiate the action (for example as described above). Forexample, in response to a first determined motion of the second vehicleand the determination that the second vehicle is a street cleaningvehicle, the first vehicle may be caused to initiate a first action; inresponse to a second determined motion of the second vehicle and thedetermination that the second vehicle is a street cleaning vehicle, thefirst vehicle may be caused to initiate a second action; and in responseto the first determined motion of the second vehicle and thedetermination that the second vehicle is not a street cleaning vehicle,the first vehicle may be caused to initiate a third action. In anotherexample, in response to a first determined motion of the second vehicleand the determination that the second vehicle is a street cleaningvehicle, the first vehicle may be caused to initiate a first action; andin response to a second determined motion of the second vehicle and thedetermination that the second vehicle is a street cleaning vehicle,causing the first vehicle to initiate the first action may be withheldand/or forwent. For example, in response to a first determined motion ofthe second vehicle and a first determined state of the street cleaningvehicle, the first vehicle may be caused to initiate a first action; inresponse to a second determined motion of the second vehicle and thefirst determined state of the street cleaning vehicle, the first vehiclemay be caused to initiate a second action; and in response to the firstdetermined motion of the second vehicle and a second determined state ofthe street cleaning vehicle, the first vehicle may be caused to initiatea third action. In another example, in response to a first determinedmotion of the second vehicle and a first determined state of the streetcleaning vehicle, the first vehicle may be caused to initiate a firstaction; in response to a second determined motion of the second vehicleand the first determined state of the street cleaning vehicle and/or inresponse to the first determined motion of the second vehicle and asecond determined state of the street cleaning vehicle, causing thefirst vehicle to initiate the first action may be withheld and/orforwent.

In some examples, it may be determining that the second vehicle issignaling, for example as described above. Further, in some examples, atype of the signaling of the second vehicle may be determined, forexample as described above. Some non-limiting examples of such type ofsignaling may include signaling using light, signaling using sound,signaling left, signaling right, signaling break, signaling hazardwarning, signaling reversing, signaling warning, and so forth. Further,in some examples, in response to the determination that the secondvehicle is signaling and/or the determined type of the signaling and/orthe determination that the second vehicle is a street cleaning vehicleand/or based on the determined state of the street cleaning vehicle, thefirst vehicle may be caused to initiate the action (for example asdescribed above). For example, in response to a first determined type ofthe signaling and the determination that the second vehicle is a streetcleaning vehicle, the first vehicle may be caused to initiate a firstaction; in response to a second determined type of the signaling and thedetermination that the second vehicle is a street cleaning vehicle, thefirst vehicle may be caused to initiate a second action; and in responseto the first determined type of the signaling and the determination thatthe second vehicle is not a street cleaning vehicle, the first vehiclemay be caused to initiate a third action. In another example, inresponse to a first determined type of the signaling and thedetermination that the second vehicle is a street cleaning vehicle, thefirst vehicle may be caused to initiate a first action; in response to asecond determined type of the signaling and the determination that thesecond vehicle is a street cleaning vehicle and/or in response to thefirst determined type of the signaling and the determination that thesecond vehicle is not a street cleaning vehicle, causing the firstvehicle to initiate the first action may be withheld and/or forwent. Forexample, in response to a first determined type of the signaling and afirst determined state of the street cleaning vehicle, the first vehiclemay be caused to initiate a first action; in response to a seconddetermined type of the signaling and the first determined state of thestreet cleaning vehicle, the first vehicle may be caused to initiate asecond action; and in response to the first determined type of thesignaling and a second determined state of the street cleaning vehicle,the first vehicle may be caused to initiate a third action. In anotherexample, in response to a first determined type of the signaling and afirst determined state of the street cleaning vehicle, the first vehiclemay be caused to initiate a first action; in response to a seconddetermined type of the signaling and the first determined state of thestreet cleaning vehicle and/or in response to the first determined typeof the signaling and a second determined state of the street cleaningvehicle, causing the first vehicle to initiate the first action may bewithheld and/or forwent. In an additional example, in response to thedetermination that the second vehicle is signaling and the determinationthat the second vehicle is a street cleaning vehicle, the first vehiclemay be caused to initiate a first action; and in response to thedetermination that the second vehicle is not signaling and thedetermination that the second vehicle is a street cleaning vehicle, thefirst vehicle may be caused to initiate a second action. In yet anotherexample, in response to the determination that the second vehicle issignaling and the determination that the second vehicle is a streetcleaning vehicle, the first vehicle may be caused to initiate a firstaction; and in response to the determination that the second vehicle isnot signaling and the determination that the second vehicle is a streetcleaning vehicle, causing the first vehicle to initiate the first actionmay be withheld and/or forwent. In an additional example, in response tothe determination that the second vehicle is signaling and a firstdetermined state of the street cleaning vehicle, the first vehicle maybe caused to initiate a first action; and in response to thedetermination that the second vehicle is not signaling and the firstdetermined state of the street cleaning vehicle, the first vehicle maybe caused to initiate a second action. In yet another example, inresponse to the determination that the second vehicle is signaling and afirst determined state of the street cleaning vehicle, the first vehiclemay be caused to initiate a first action; and in response to thedetermination that the second vehicle is not signaling and the firstdetermined state of the street cleaning vehicle, causing the firstvehicle to initiate the first action may be withheld and/or forwent.

In some examples, a position of the second vehicle may be determined,for example as described above. For example, the determined position maybe in relation to the ground, in relation to a map, in relation to aroad, in relation to a lane, in relation to an object in theenvironment, in relation to the at least part of the first vehicle, inrelation to the image sensor, and so forth. Further, in some examples,it may be determining that the second vehicle is in a lane of the firstvehicle, for example as described above. Further, in some examples, itmay be determining that the second vehicle is in a planned path of thefirst vehicle, for example as described above. Further, in someexamples, in response to the determined position of the second vehicleand/or the determination that the second vehicle is a street cleaningvehicle and/or based the first determined state of the street cleaningvehicle, the first vehicle may be caused to initiate the action (forexample as described above). For example, in response to a firstdetermined position of the second vehicle and the determination that thesecond vehicle is a street cleaning vehicle, the first vehicle may becaused to initiate a first action; in response to a second determinedposition of the second vehicle and the determination that the secondvehicle is a street cleaning vehicle, the first vehicle may be caused toinitiate a second action; and in response to the first determinedposition of the second vehicle and the determination that the secondvehicle is not a street cleaning vehicle, the first vehicle may becaused to initiate a third action. In another example, in response to afirst determined position of the second vehicle and the determinationthat the second vehicle is a street cleaning vehicle, the first vehiclemay be caused to initiate a first action; in response to a seconddetermined position of the second vehicle and the determination that thesecond vehicle is a street cleaning vehicle and/or in response to thefirst determined position of the second vehicle and the determinationthat the second vehicle is not a street cleaning vehicle, causing thefirst vehicle to initiate the first action may be withheld and/orforwent. In an additional example, in response to the determination thatthe second vehicle is in a lane and/or a planned path of the firstvehicle and the determination that the second vehicle is a streetcleaning vehicle, the first vehicle may be caused to initiate a firstaction; and in response to the determination that the second vehicle isnot in a lane and/or is not in a planned path of the first vehicle andthe determination that the second vehicle is a street cleaning vehicle,the first vehicle may be caused to initiate a second action. In yetanother example, in response to the determination that the secondvehicle is in a lane and/or a planned path of the first vehicle and thedetermination that the second vehicle is a street cleaning vehicle, thefirst vehicle may be caused to initiate a first action; and in responseto the determination that the second vehicle is not in a lane and/or isnot in a planned path of the first vehicle and the determination thatthe second vehicle is a street cleaning vehicle, causing the firstvehicle to initiate the first action may be withheld and/or forwent. Forexample, in response to a first determined position of the secondvehicle and a first determined state of the street cleaning vehicle, thefirst vehicle may be caused to initiate a first action; in response to asecond determined position of the second vehicle and the firstdetermined state of the street cleaning vehicle, the first vehicle maybe caused to initiate a second action; and in response to the firstdetermined position of the second vehicle and a second determined stateof the street cleaning vehicle, the first vehicle may be caused toinitiate a third action. In another example, in response to a firstdetermined position of the second vehicle and a first determined stateof the street cleaning vehicle, the first vehicle may be caused toinitiate a first action; in response to a second determined position ofthe second vehicle and the first determined state of the street cleaningvehicle and/or in response to the first determined position of thesecond vehicle and a second determined state of the street cleaningvehicle, causing the first vehicle to initiate the first action may bewithheld and/or forwent. In an additional example, in response to thedetermination that the second vehicle is in a lane and/or a planned pathof the first vehicle and a first determined state of the street cleaningvehicle, the first vehicle may be caused to initiate a first action; andin response to the determination that the second vehicle is not in alane and/or is not in a planned path of the first vehicle and the firstdetermined state of the street cleaning vehicle, the first vehicle maybe caused to initiate a second action. In yet another example, inresponse to the determination that the second vehicle is in a laneand/or a planned path of the first vehicle and a first determined stateof the street cleaning vehicle, the first vehicle may be caused toinitiate a first action; and in response to the determination that thesecond vehicle is not in a lane and/or is not in a planned path of thefirst vehicle and the first determined state of the street cleaningvehicle, causing the first vehicle to initiate the first action may bewithheld and/or forwent.

In some embodiments, systems, methods and computer readable media forcontrolling vehicles based on hoods of other vehicles are provided. Onechallenge of autonomous driving is to determine when a standing vehiclehas stopped for a short period of time and therefore the autonomousvehicle needs to wait for the standing vehicle to resume moving, andwhen the standing vehicle has stopped for a longer period of time andtherefore the autonomous vehicle needs to pass the standing vehicle. Insome cases, one indication of whether a vehicle with a hood has stoppedfor a longer period of time may include the state of the hood. Forexample, identifying that the hood is open may indicate that the vehiclewith the hood has stopped for a longer period of time. The providedsystems, methods and computer readable media for controlling vehiclesmay detect a vehicle connected to a hood, identify a state of the hood,and control a response of an autonomous vehicle to the detected vehiclebased on the identified state of the hood.

FIG. 17 illustrates an example of a method 1700 for controlling vehiclesbased on hoods of other vehicles. In this example, method 1700 maycomprise: obtaining images captured from an environment of a firstvehicle (Step 710); analyzing the images to detect a second vehicle(Step 720); analyzing the images to determine a state of a hood of thesecond vehicle (Step 1730); in response to a first determined state ofthe hood, causing the first vehicle to initiate an action responding tothe second vehicle (Step 1740), and in response to a second determinedstate of the hood, forgoing causing the first vehicle to initiate theaction. In some implementations, method 1700 may comprise one or moreadditional steps, while some of the steps listed above may be modifiedor excluded. In some implementations, one or more steps illustrated inFIG. 17 may be executed in a different order and/or one or more groupsof steps may be executed simultaneously and/or a plurality of steps maybe combined into single step and/or a single step may be broken down toa plurality of steps.

In some embodiments, one or more images captured using one or more imagesensors from an environment of a first vehicle may be obtained, forexample as described above. Further, in some examples, the one or moreimages may be analyzed to detect a second vehicle, for example asdescribed above. Further, in some examples, the one or more images maybe analyzed to determine a state of a hood of the second vehicle.Further, in some examples, in response to a first determined state ofthe hood of the second vehicle, the first vehicle may be caused toinitiate an action responding to the second vehicle.

In some embodiments, step 1730 may comprise analyzing the one or moreimages obtained by step 710 to determine a state of the hood of thesecond vehicle detected by step 720. Some non-limiting examples of suchstate of the hood of the second vehicle may include hood of the secondvehicle in open, hood of the second vehicle not in close, hood of thesecond vehicle completely open, hood of the second vehicle substantiallyclosed, hood of the second vehicle is opening, hood of the secondvehicle is closing, and so forth. For example, a machine learning modelmay be trained using training examples to determine states of hoods ofvehicles from images and/or videos, and step 1730 may use the trainedmachine learning model to analyze the one or more images obtained bystep 710 and determine the state of the hood of the second vehicledetected by step 720. An example of such training example may include animage and/or a video depicting a hood of a vehicle, together with alabel indicating the state of the depicted hood. In another example, anartificial neural network (such as a deep neural networks, convolutionalneural networks, etc.) may be configured to determine states of hoods ofvehicles from images and/or videos, and step 1730 may use the artificialneural network to analyze the one or more images obtained by step 710and determine the state of the hood of the second vehicle detected bystep 720. In yet another example, step 1730 may use image classificationalgorithms to analyze the one or more images obtained by step 710 anddetermine the state of the hood of the second vehicle detected by step720. Some other non-limiting examples of steps that may be used by step1730 for determining the state of the hood of the second vehicle aredescribed below.

In some examples, an orientation of at least part of the hood of thesecond vehicle (in relation to at least one other part of the secondvehicle, in relation to the ground, in relation to the horizon, inrelation to an object in the environment, in relation to the at leastpart of the first vehicle, in relation to the image sensor, etc.) may bedetermined, and the determined state of the hood of the second vehiclemay be based on the determined orientation of the at least part of thehood of the second vehicle. For example, a machine learning model may betrained using training examples to determine orientation of parts ofhoods by analyzing images and/or videos, and the trained machinelearning model may be used to determine the orientation of the at leastpart of the hood of the second vehicle from the one or more images. Inanother example, an artificial neural network (such as a deep neuralnetworks, convolutional neural networks, etc.) may be configured todetermine orientation of parts of hoods by analyzing images and/orvideos, and the configured artificial neural network may be used todetermine the orientation of the at least part of the hood of the secondvehicle from the one or more images. In yet another example, informationabout the orientation of the at least part of the hood of the secondvehicle may be received from the second vehicle (for example, using apoint to point communication protocol, through a communication networkusing a communication device, through a centralized server, and soforth).

In some examples, a distance of at least part of the hood of the secondvehicle (from at least part of the second vehicle, from the ground, froman object in the environment, from at least part of the first vehicle,from the image sensor, etc.) may be determined, and the determined stateof the hood of the second vehicle may be based on the determineddistance of the at least part of the hood of the second vehicle. Forexample, a machine learning model may be trained using training examplesto determine distance of parts of hoods by analyzing images and/orvideos, and the trained machine learning model may be used to determinethe distance of the at least part of the hood of the second vehicle fromthe one or more images. In another example, an artificial neural network(such as a deep neural networks, convolutional neural networks, etc.)may be configured to determine distance of parts of hoods by analyzingimages and/or videos, and the configured artificial neural network maybe used to determine the distance of the at least part of the hood ofthe second vehicle from the one or more images. In yet another example,information about the distance of the at least part of the hood of thesecond vehicle may be received from the second vehicle (for example,using a point to point communication protocol, through a communicationnetwork using a communication device, through a centralized server, andso forth).

In some examples, a motion of at least part of the hood of the secondvehicle (in relation to at least one other part of the second vehicle,in relation to the ground, in relation to the horizon, in relation to anobject in the environment, in relation to the at least part of the firstvehicle, in relation to the image sensor, etc.) may be determined, andthe determined state of the hood of the second vehicle may be based onthe determined motion of the at least part of the hood of the secondvehicle. For example, a machine learning model may be trained usingtraining examples to determine motion of parts of hoods by analyzingimages and/or videos, and the trained machine learning model may be usedto determine the motion of the at least part of the hood of the secondvehicle from the one or more images. In another example, an artificialneural network (such as a deep neural networks, convolutional neuralnetworks, etc.) may be configured to determine motion of parts of hoodsby analyzing images and/or videos, and the configured artificial neuralnetwork may be used to determine the motion of the at least part of thehood of the second vehicle from the one or more images. In yet anotherexample, information about the motion of the at least part of the hoodof the second vehicle may be received from the second vehicle (forexample, using a point to point communication protocol, through acommunication network using a communication device, through acentralized server, and so forth).

In some embodiments, step 1740 may comprise, for example in response toa first determined state of the hood of the second vehicle detected bystep 720 (e.g., as determined by step 1730), causing the first vehicleto initiate an action responding to the second vehicle detected by step720. Some non-limiting examples of such action may include signaling,stopping, changing a speed of the first vehicle, changing a motiondirection of the first vehicle, passing the second vehicle, forgoingpassing the second vehicle, keeping a minimal distance of at least aselected length from the second vehicle (for example, the selectedlength may be less than 100 feet, may be at least 100 feet, may be atleast 200 feet, etc.), turning, performing a U-turn, driving in reverse,generating an audible warning, and so forth. For example, step 1740 maytransmit a signal to an external device (such as a device controllingthe first vehicle, a device navigating the first vehicle, the firstvehicle, a system within the first vehicle, etc.), and the signal may beconfigured to cause the external device to cause the first vehicle toinitiate the action responding to the second vehicle detected by step720. In another example, step 1740 may provide information related tothe second vehicle to such external device, and the information may beconfigured to cause the external device to cause the first vehicle toinitiate the action responding to the second vehicle detected by step720. In one example, in response to a second state of the hood of thesecond vehicle detected by step 720 (e.g., as determined by step 1730),step 1740 may cause the first vehicle to initiate a second action, thesecond action may differ from the action.

In some examples, step 1740 may select the action based on thedetermined state of the hood of the second vehicle detected by step 720(e.g., as determined by step 1730). For example, in response to a firststate of the hood of the second vehicle detected by step 720 (e.g., asdetermined by step 1730), step 1740 may select a first action, and inresponse to a second state of the hood of the second vehicle detected bystep 720 (e.g., as determined by step 1730), step 1740 may select asecond action (where the second action may differ from the first action,and the second state of the hood of the second vehicle may differ fromthe first state of the hood of the second vehicle). In some examples, inresponse to a first state of the hood of the second vehicle detected bystep 720 (e.g., as determined by step 1730), step 1740 may select afirst action, and in response to a second state of the hood of thesecond vehicle detected by step 720 (e.g., as determined by step 1730),step 1740 may withhold and/or forgo the first action (for example, step1740 may withhold and/or forgo causing the first vehicle to initiate theaction).

In some examples, a motion of the second vehicle may be determined, forexample as described above. Further, in some examples, in response tothe determined motion of the second vehicle and the first determinedstate of the hood of the second vehicle, the first vehicle may be causedto initiate the action (for example as described above). For example, inresponse to a first determined motion of the second vehicle and a firstdetermined state of the hood of the second vehicle, the first vehiclemay be caused to initiate a first action; in response to a seconddetermined motion of the second vehicle and the first determined stateof the hood of the second vehicle, the first vehicle may be caused toinitiate a second action; and in response to the first determined motionof the second vehicle and a second determined state of the hood of thesecond vehicle, the first vehicle may be caused to initiate a thirdaction. In another example, in response to a first determined motion ofthe second vehicle and a first determined state of the hood of thesecond vehicle, the first vehicle may be caused to initiate a firstaction; in response to a second determined motion of the second vehicleand the first determined state of the hood of the second vehicle and/orin response to the first determined motion of the second vehicle and asecond determined state of the hood of the second vehicle, causing thefirst vehicle to initiate the first action may be withheld and/orforwent.

In some examples, it may be determining that the second vehicle issignaling, for example as described above. Further, in some examples, atype of the signaling of the second vehicle may be determined, forexample as described above. Some non-limiting examples of such type ofsignaling may include signaling using light, signaling using sound,signaling left, signaling right, signaling break, signaling hazardwarning, signaling reversing, signaling warning, and so forth. Further,in some examples, in response to the determination that the secondvehicle is signaling and/or the determined type of the signaling and/orthe first determined state of the hood of the second vehicle, the firstvehicle may be caused to initiate the action (for example as describedabove). For example, in response to a first determined type of thesignaling and a first determined state of the hood of the secondvehicle, the first vehicle may be caused to initiate a first action; inresponse to a second determined type of the signaling and the firstdetermined state of the hood of the second vehicle, the first vehiclemay be caused to initiate a second action; and in response to the firstdetermined type of the signaling and a second determined state of thehood of the second vehicle, the first vehicle may be caused to initiatea third action. In another example, in response to a first determinedtype of the signaling and a first determined state of the hood of thesecond vehicle, the first vehicle may be caused to initiate a firstaction; in response to a second determined type of the signaling and thefirst determined state of the hood of the second vehicle and/or inresponse to the first determined type of the signaling and a seconddetermined state of the hood of the second vehicle, causing the firstvehicle to initiate the first action may be withheld and/or forwent. Inan additional example, in response to the determination that the secondvehicle is signaling and a first determined state of the hood of thesecond vehicle, the first vehicle may be caused to initiate a firstaction; and in response to the determination that the second vehicle isnot signaling and the first determined state of the hood of the secondvehicle, the first vehicle may be caused to initiate a second action. Inyet another example, in response to the determination that the secondvehicle is signaling and a first determined state of the hood of thesecond vehicle, the first vehicle may be caused to initiate a firstaction; and in response to the determination that the second vehicle isnot signaling and the first determined state of the hood of the secondvehicle, causing the first vehicle to initiate the first action may bewithheld and/or forwent.

In some examples, a position of the second vehicle may be determined,for example as described above. For example, the determined position maybe in relation to the ground, in relation to a map, in relation to aroad, in relation to a lane, in relation to an object in theenvironment, in relation to the at least part of the first vehicle, inrelation to the image sensor, and so forth. Further, in some examples,it may be determining that the second vehicle is in a lane of the firstvehicle, for example as described above. Further, in some examples, itmay be determining that the second vehicle is in a planned path of thefirst vehicle, for example as described above. Further, in someexamples, in response to the determined position of the second vehicleand the first determined state of the hood of the second vehicle, thefirst vehicle may be caused to initiate the action (for example asdescribed above). For example, in response to a first determinedposition of the second vehicle and a first determined state of the hoodof the second vehicle, the first vehicle may be caused to initiate afirst action; in response to a second determined position of the secondvehicle and the first determined state of the hood of the secondvehicle, the first vehicle may be caused to initiate a second action;and in response to the first determined position of the second vehicleand a second determined state of the hood of the second vehicle, thefirst vehicle may be caused to initiate a third action. In anotherexample, in response to a first determined position of the secondvehicle and a first determined state of the hood of the second vehicle,the first vehicle may be caused to initiate a first action; in responseto a second determined position of the second vehicle and the firstdetermined state of the hood of the second vehicle and/or in response tothe first determined position of the second vehicle and a seconddetermined state of the hood of the second vehicle, causing the firstvehicle to initiate the first action may be withheld and/or forwent. Inan additional example, in response to the determination that the secondvehicle is in a lane and/or a planned path of the first vehicle and afirst determined state of the hood of the second vehicle, the firstvehicle may be caused to initiate a first action; and in response to thedetermination that the second vehicle is not in a lane and/or is not ina planned path of the first vehicle and the first determined state ofthe hood of the second vehicle, the first vehicle may be caused toinitiate a second action. In yet another example, in response to thedetermination that the second vehicle is in a lane and/or a planned pathof the first vehicle and a first determined state of the hood of thesecond vehicle, the first vehicle may be caused to initiate a firstaction; and in response to the determination that the second vehicle isnot in a lane and/or is not in a planned path of the first vehicle andthe first determined state of the hood of the second vehicle, causingthe first vehicle to initiate the first action may be withheld and/orforwent.

In some embodiments, systems, methods and computer readable media forcontrolling vehicles based on trunk lids of other vehicles are provided.One challenge of autonomous driving is to determine when a standingvehicle has stopped for a short period of time and therefore theautonomous vehicle needs to wait for the standing vehicle to resumemoving, and when the standing vehicle has stopped for a longer period oftime and therefore the autonomous vehicle needs to pass the standingvehicle. In some cases, one indication of whether a vehicle with a trunklid has stopped for a longer period of time may include the state of thetrunk lid. For example, identifying that the trunk lid is open mayindicate that the vehicle with the trunk lid has stopped for a longerperiod of time. The provided systems, methods and computer readablemedia for controlling vehicles may detect a vehicle connected to a trunklid, identify a state of the trunk lid, and control a response of anautonomous vehicle to the detected vehicle based on the identified stateof the trunk lid.

FIG. 18 illustrates an example of a method 1800 for controlling vehiclesbased on trunk lids of other vehicles. In this example, method 1800 maycomprise: obtaining images captured from an environment of a firstvehicle (Step 710); analyzing the images to detect a second vehicle(Step 720); analyzing the images to determine a state of a trunk lid ofthe second vehicle (Step 1830); in response to a first determined stateof the trunk lid, causing the first vehicle to initiate an actionresponding to the second vehicle (Step 1840), and in response to asecond determined state of the trunk lid, forgoing causing the firstvehicle to initiate the action. In some implementations, method 1800 maycomprise one or more additional steps, while some of the steps listedabove may be modified or excluded. In some implementations, one or moresteps illustrated in FIG. 18 may be executed in a different order and/orone or more groups of steps may be executed simultaneously and/or aplurality of steps may be combined into single step and/or a single stepmay be broken down to a plurality of steps.

In some embodiments, one or more images captured using one or more imagesensors from an environment of a first vehicle may be obtained, forexample as described above. Further, in some examples, the one or moreimages may be analyzed to detect a second vehicle, for example asdescribed above. Further, in some examples, the one or more images maybe analyzed to determine a state of a trunk lid of the second vehicle.Further, in some examples, in response to a first determined state ofthe trunk lid of the second vehicle, the first vehicle may be caused toinitiate an action responding to the second vehicle.

In some embodiments, step 1830 may comprise analyzing the one or moreimages obtained by step 710 to determine a state of the trunk lid of thesecond vehicle detected by step 720. Some non-limiting examples of suchstate of the trunk lid of the second vehicle may include trunk lid ofthe second vehicle in open, trunk lid of the second vehicle not inclose, trunk lid of the second vehicle completely open, trunk lid of thesecond vehicle substantially closed, trunk lid of the second vehicle isopening, trunk lid of the second vehicle is closing, and so forth. Forexample, a machine learning model may be trained using training examplesto determine states of trunk lids of vehicles by analyzing images and/orvideos, and step 1830 may use the trained machine learning model toanalyze the one or more images obtained by step 710 and determine thestate of the trunk lid of the second vehicle detected by step 720. Anexample of such training example may include an image and/or a videodepicting a vehicle, together with a label indicating the state of atrunk lid of the depicted vehicle. In another example, an artificialneural network (such as a deep neural networks, convolutional neuralnetworks, etc.) may be configured to determine states of trunk lids ofvehicles by analyzing images and/or videos, and step 1830 may use theartificial neural network to analyze the one or more images obtained bystep 710 and determine the state of the trunk lid of the second vehicledetected by step 720. In yet another example, step 1830 may use imageclassification algorithms to analyze the one or more images obtained bystep 710 and determine the state of the trunk lid of the second vehicle.Some other non-limiting examples of steps that may be used by step 1830for determining the state of the trunk lid of the second vehicle aredescribed below.

In some examples, an orientation of at least part of the trunk lid ofthe second vehicle (in relation to at least one other part of the secondvehicle, in relation to the ground, in relation to the horizon, inrelation to an object in the environment, in relation to the at leastpart of the first vehicle, in relation to the image sensor, etc.) may bedetermined, and the determined state of the trunk lid of the secondvehicle may be based on the determined orientation of the at least partof the trunk lid of the second vehicle. For example, a machine learningmodel may be trained using training examples to determine orientation ofparts of trunk lids by analyzing images and/or videos, and the trainedmachine learning model may be used to determine the orientation of theat least part of the trunk lid of the second vehicle from the one ormore images. In another example, an artificial neural network (such as adeep neural networks, convolutional neural networks, etc.) may beconfigured to determine orientation of parts of trunk lids by analyzingimages and/or videos, and the configured artificial neural network maybe used to determine the orientation of the at least part of the trunklid of the second vehicle from the one or more images. In yet anotherexample, information about the orientation of the at least part of thetrunk lid of the second vehicle may be received from the second vehicle(for example, using a point to point communication protocol, through acommunication network using a communication device, through acentralized server, and so forth).

In some examples, a distance of at least part of the trunk lid of thesecond vehicle (from at least part of the second vehicle, from theground, from an object in the environment, from at least part of thefirst vehicle, from the image sensor, etc.) may be determined, and thedetermined state of the trunk lid of the second vehicle may be based onthe determined distance of the at least part of the trunk lid of thesecond vehicle. For example, a machine learning model may be trainedusing training examples to determine distance of parts of trunk lids byanalyzing images and/or videos, and the trained machine learning modelmay be used to determine the distance of the at least part of the trunklid of the second vehicle from the one or more images. In anotherexample, an artificial neural network (such as a deep neural networks,convolutional neural networks, etc.) may be configured to determinedistance of parts of trunk lids by analyzing images and/or videos, andthe configured artificial neural network may be used to determine thedistance of the at least part of the trunk lid of the second vehiclefrom the one or more images. In yet another example, information aboutthe distance of the at least part of the trunk lid of the second vehiclemay be received from the second vehicle (for example, using a point topoint communication protocol, through a communication network using acommunication device, through a centralized server, and so forth).

In some examples, a motion of at least part of the trunk lid of thesecond vehicle (in relation to at least one other part of the secondvehicle, in relation to the ground, in relation to the horizon, inrelation to an object in the environment, in relation to the at leastpart of the first vehicle, in relation to the image sensor, etc.) may bedetermined, and the determined state of the trunk lid of the secondvehicle may be based on the determined motion of the at least part ofthe trunk lid of the second vehicle. For example, a machine learningmodel may be trained using training examples to determine motion ofparts of trunk lids by analyzing images and/or videos, and the trainedmachine learning model may be used to determine the motion of the atleast part of the trunk lid of the second vehicle from the one or moreimages. In another example, an artificial neural network (such as a deepneural networks, convolutional neural networks, etc.) may be configuredto determine motion of parts of trunk lids by analyzing images and/orvideos, and the configured artificial neural network may be used todetermine the motion of the at least part of the trunk lid of the secondvehicle from the one or more images. In yet another example, informationabout the motion of the at least part of the trunk lid of the secondvehicle may be received from the second vehicle (for example, using apoint to point communication protocol, through a communication networkusing a communication device, through a centralized server, and soforth).

In some embodiments, step 1840 may comprise, for example in response toa first determined state of the trunk lid of the second vehicle detectedby step 720 (e.g., as determined by step 1830), causing the firstvehicle to initiate an action responding to the second vehicle detectedby step 720. Some non-limiting examples of such action may includesignaling, stopping, changing a speed of the first vehicle, changing amotion direction of the first vehicle, passing the second vehicle,forgoing passing the second vehicle, keeping a minimal distance of atleast a selected length from the second vehicle (for example, theselected length may be less than 100 feet, may be at least 100 feet, maybe at least 200 feet, etc.), turning, performing a U-turn, driving inreverse, generating an audible warning, and so forth. For example, step1840 may transmit a signal to an external device (such as a devicecontrolling the first vehicle, a device navigating the first vehicle,the first vehicle, a system within the first vehicle, etc.), and thesignal may be configured to cause the external device to cause the firstvehicle to initiate the action responding to the second vehicle detectedby step 720. In another example, step 1840 may provide informationrelated to the second vehicle to such external device, and theinformation may be configured to cause the external device to cause thefirst vehicle to initiate the action responding to the second vehicledetected by step 720. In one example, in response to a second state ofthe trunk lid of the second vehicle detected by step 720 (e.g., asdetermined by step 1830), step 1840 may cause the first vehicle toinitiate a second action, the second action may differ from the action.

In some examples, step 1840 may select the action based on thedetermined state of the trunk lid of the second vehicle detected by step720 (e.g., as determined by step 1830). For example, in response to afirst state of the trunk lid of the second vehicle detected by step 720(e.g., as determined by step 1830), step 1840 may select a first action,and in response to a second state of the trunk lid of the second vehicledetected by step 720 (e.g., as determined by step 1830), step 1840 mayselect a second action (where the second action may differ from thefirst action, and the second state of the trunk lid of the secondvehicle may differ from the first state of the trunk lid of the secondvehicle). In some examples, in response to a first state of the trunklid of the second vehicle detected by step 720 (e.g., as determined bystep 1830), step 1840 may select a first action, and in response to asecond state of the trunk lid of the second vehicle detected by step 720(e.g., as determined by step 1830), step 1840 may withhold and/or forgothe first action (for example, step 1840 may withhold and/or forgocausing the first vehicle to initiate the action).

In some examples, a motion of the second vehicle may be determined, forexample as described above. Further, in some examples, in response tothe determined motion of the second vehicle and the first determinedstate of the trunk lid of the second vehicle, the first vehicle may becaused to initiate the action (for example as described above). Forexample, in response to a first determined motion of the second vehicleand a first determined state of the trunk lid of the second vehicle, thefirst vehicle may be caused to initiate a first action; in response to asecond determined motion of the second vehicle and the first determinedstate of the trunk lid of the second vehicle, the first vehicle may becaused to initiate a second action; and in response to the firstdetermined motion of the second vehicle and a second determined state ofthe trunk lid of the second vehicle, the first vehicle may be caused toinitiate a third action. In another example, in response to a firstdetermined motion of the second vehicle and a first determined state ofthe trunk lid of the second vehicle, the first vehicle may be caused toinitiate a first action; in response to a second determined motion ofthe second vehicle and the first determined state of the trunk lid ofthe second vehicle and/or in response to the first determined motion ofthe second vehicle and a second determined state of the trunk lid of thesecond vehicle, causing the first vehicle to initiate the first actionmay be withheld and/or forwent.

In some examples, it may be determining that the second vehicle issignaling, for example as described above. Further, in some examples, atype of the signaling of the second vehicle may be determined, forexample as described above. Some non-limiting examples of such type ofsignaling may include signaling using light, signaling using sound,signaling left, signaling right, signaling break, signaling hazardwarning, signaling reversing, signaling warning, and so forth. Further,in some examples, in response to the determination that the secondvehicle is signaling and/or the determined type of the signaling and/orthe first determined state of the trunk lid of the second vehicle, thefirst vehicle may be caused to initiate the action (for example asdescribed above). For example, in response to a first determined type ofthe signaling and a first determined state of the trunk lid of thesecond vehicle, the first vehicle may be caused to initiate a firstaction; in response to a second determined type of the signaling and thefirst determined state of the trunk lid of the second vehicle, the firstvehicle may be caused to initiate a second action; and in response tothe first determined type of the signaling and a second determined stateof the trunk lid of the second vehicle, the first vehicle may be causedto initiate a third action. In another example, in response to a firstdetermined type of the signaling and a first determined state of thetrunk lid of the second vehicle, the first vehicle may be caused toinitiate a first action; in response to a second determined type of thesignaling and the first determined state of the trunk lid of the secondvehicle and/or in response to the first determined type of the signalingand a second determined state of the trunk lid of the second vehicle,causing the first vehicle to initiate the first action may be withheldand/or forwent. In an additional example, in response to thedetermination that the second vehicle is signaling and a firstdetermined state of the trunk lid of the second vehicle, the firstvehicle may be caused to initiate a first action; and in response to thedetermination that the second vehicle is not signaling and the firstdetermined state of the trunk lid of the second vehicle, the firstvehicle may be caused to initiate a second action. In yet anotherexample, in response to the determination that the second vehicle issignaling and a first determined state of the trunk lid of the secondvehicle, the first vehicle may be caused to initiate a first action; andin response to the determination that the second vehicle is notsignaling and the first determined state of the trunk lid of the secondvehicle, causing the first vehicle to initiate the first action may bewithheld and/or forwent.

In some examples, a position of the second vehicle may be determined,for example as described above. For example, the determined position maybe in relation to the ground, in relation to a map, in relation to aroad, in relation to a lane, in relation to an object in theenvironment, in relation to the at least part of the first vehicle, inrelation to the image sensor, and so forth. Further, in some examples,it may be determining that the second vehicle is in a lane of the firstvehicle, for example as described above. Further, in some examples, itmay be determining that the second vehicle is in a planned path of thefirst vehicle, for example as described above. Further, in someexamples, in response to the determined position of the second vehicleand the first determined state of the trunk lid of the second vehicle,the first vehicle may be caused to initiate the action (for example asdescribed above). For example, in response to a first determinedposition of the second vehicle and a first determined state of the trunklid of the second vehicle, the first vehicle may be caused to initiate afirst action; in response to a second determined position of the secondvehicle and the first determined state of the trunk lid of the secondvehicle, the first vehicle may be caused to initiate a second action;and in response to the first determined position of the second vehicleand a second determined state of the trunk lid of the second vehicle,the first vehicle may be caused to initiate a third action. In anotherexample, in response to a first determined position of the secondvehicle and a first determined state of the trunk lid of the secondvehicle, the first vehicle may be caused to initiate a first action; inresponse to a second determined position of the second vehicle and thefirst determined state of the trunk lid of the second vehicle and/or inresponse to the first determined position of the second vehicle and asecond determined state of the trunk lid of the second vehicle, causingthe first vehicle to initiate the first action may be withheld and/orforwent. In an additional example, in response to the determination thatthe second vehicle is in a lane and/or a planned path of the firstvehicle and a first determined state of the trunk lid of the secondvehicle, the first vehicle may be caused to initiate a first action; andin response to the determination that the second vehicle is not in alane and/or is not in a planned path of the first vehicle and the firstdetermined state of the trunk lid of the second vehicle, the firstvehicle may be caused to initiate a second action. In yet anotherexample, in response to the determination that the second vehicle is ina lane and/or a planned path of the first vehicle and a first determinedstate of the trunk lid of the second vehicle, the first vehicle may becaused to initiate a first action; and in response to the determinationthat the second vehicle is not in a lane and/or is not in a planned pathof the first vehicle and the first determined state of the trunk lid ofthe second vehicle, causing the first vehicle to initiate the firstaction may be withheld and/or forwent.

In some embodiments, systems, methods and computer readable media forcontrolling vehicles in response to smoke are provided. One challenge ofautonomous driving is to determine when a standing vehicle has stoppedfor a short period of time and therefore the autonomous vehicle needs towait for the standing vehicle to resume moving, and when the standingvehicle has stopped for a longer period of time and therefore theautonomous vehicle needs to pass the standing vehicle. In some cases,identifying that unusual emission of gases (such as smoke, steam, etc.)may indicate that the vehicle has stopped for a longer period of time.The provided systems, methods and computer readable media forcontrolling vehicles may detect a vehicle, identify whether the vehicleemits unusual smoke, and control a response of an autonomous vehicle tothe detected vehicle based on whether the vehicle emits unusual smoke.

FIG. 19 illustrates an example of a method 1900 for controlling vehiclesin response to smoke. In this example, method 1900 may comprise:obtaining images captured from an environment of a first vehicle (Step710); analyzing the images to detect a second vehicle (Step 720);analyzing the images to determine that the second vehicle emits smokenot through an exhaust system (Step 1930); in response to thedetermination that the second vehicle emits smoke not through an exhaustsystem, causing the first vehicle to initiate an action responding tothe second vehicle (Step 1940), and in response to a determination thatthe second vehicle does not emit smoke or emits smoke only through anexhaust system, forgoing causing the first vehicle to initiate theaction. In some implementations, method 1900 may comprise one or moreadditional steps, while some of the steps listed above may be modifiedor excluded. In some implementations, one or more steps illustrated inFIG. 19 may be executed in a different order and/or one or more groupsof steps may be executed simultaneously and/or a plurality of steps maybe combined into single step and/or a single step may be broken down toa plurality of steps.

It is to be understood that any embodiment and/or example related tosmoke in this disclosure may also be similarly implemented with relationto steam, to a combination of smoke and steam, other visible gases, andso forth.

In some embodiments, one or more images captured using one or more imagesensors from an environment of a first vehicle may be obtained, forexample as described above. Further, in some examples, the one or moreimages may be analyzed to detect a second vehicle, for example asdescribed above. Further, in some examples, the one or more images maybe analyzed to determine whether the second vehicle emits smoke notthrough an exhaust system. Further, in examples, for example in responseto the determination that the second vehicle emits smoke not through anexhaust system, the first vehicle may be caused to initiate an actionresponding to the second vehicle, for example as described above.

In some embodiments, step 1930 may comprise analyzing the one or moreimages obtained by step 710 to determine whether the second vehicledetected by step 720 emits smoke not through an exhaust system. Forexample, a machine learning model may be trained using training examplesto determine whether vehicles are emitting smoke not through exhaustsystems by analyzing images and/or videos, and step 1930 may use thetrained machine learning model to analyze the one or more imagesobtained by step 710 and determine whether the second vehicle detectedby step 720 emits smoke not through an exhaust system. An example ofsuch training example may include an image and/or a video depicting avehicle, together with a label indicating whether the depicted vehicleemits smoke not through an exhaust system. In another example, anartificial neural network (such as a deep neural networks, convolutionalneural networks, etc.) may be configured to determine whether vehiclesare emitting smoke not through exhaust systems by analyzing imagesand/or videos, and step 1930 may use the artificial neural network toanalyze the one or more images obtained by step 710 and determinewhether the second vehicle detected by step 720 emits smoke not throughan exhaust system. In yet another example, step 1930 may use imageclassification algorithms to analyze the one or more images obtained bystep 710 and determine whether the second vehicle detected by step 720emits smoke not through an exhaust system.

In some embodiments, step 1940 may comprise, for example in response toa determination by step 1930 that the second vehicle detected by step720 emits smoke not through an exhaust system, causing the first vehicleto initiate an action responding to the second vehicle. Somenon-limiting examples of such action may include signaling, stopping,changing a speed of the first vehicle, changing a motion direction ofthe first vehicle, passing the second vehicle, forgoing passing thesecond vehicle, keeping a minimal distance of at least a selected lengthfrom the second vehicle (for example, the selected length may be lessthan 100 feet, may be at least 100 feet, may be at least 200 feet,etc.), turning, performing a U-turn, driving in reverse, generating anaudible warning, and so forth. For example, step 1940 may transmit asignal to an external device (such as a device controlling the firstvehicle, a device navigating the first vehicle, the first vehicle, asystem within the first vehicle, etc.), and the signal may be configuredto cause the external device to cause the first vehicle to initiate theaction responding to the second vehicle detected by step 720. In anotherexample, step 1940 may provide information related to the second vehicleto such external device, and the information may be configured to causethe external device to cause the first vehicle to initiate the actionresponding to the second vehicle detected by step 720. In one example,in response to a determination by step 1930 that the second vehicledetected by step 720 emits smoke only through an exhaust system or doesnot emit smoke at all, step 1940 may cause the first vehicle to initiatea second action, the second action may differ from the action. In oneexample, in response to a determination by step 1930 that the secondvehicle detected by step 720 emits smoke not through an exhaust system,step 1940 may cause the first vehicle to initiate a first action, and inresponse to a determination by step 1930 that the second vehicledetected by step 720 emits smoke only through an exhaust system or doesnot emit smoke at all, step 1940 may cause the first vehicle to initiatea second action, the second action may differ from the first action. Inanother example, in response to a determination by step 1930 that thesecond vehicle detected by step 720 emits smoke not through an exhaustsystem, step 1940 may cause the first vehicle to initiate a firstaction, and in response to a determination by step 1930 that the secondvehicle detected by step 720 emits smoke only through an exhaust systemor does not emit smoke at all, step 1940 may withhold and/or forgocausing the first vehicle to initiate the first action.

In some examples, step 1940 may select the action based on determinedproperties of the emission of the smoke and/or based on determinedproperties of the emitted smoke. Some non-limiting examples of suchproperties are described below. For example, in response to a firstdetermined property, step 1940 may select a first action, and inresponse to a second determined property, step 1940 may select a secondaction, the second action may differ from the first action. In someexamples, in response to a first determined property, step 1940 mayselect a first action, and in response to a second determined property,step 1940 may withhold and/or forgo the first action (for example, step1940 may withhold and/or forgo causing the first vehicle to initiate theaction).

In some embodiments, the one or more images obtained by step 710 may beanalyzed to detect smoke, for example as described below. Further, insome examples, the one or more images obtained by step 710 may beanalyzed to determine whether the detected smoke is associated with afirst side of the second vehicle detected by step 720 (such as the frontside of the second vehicle), a second side of the second vehicledetected by step 720 (such as the back side of the second vehicle), ornot associated with the second vehicle detected by step 720 (for exampleas described below). Further, in some examples, in response to adetermination that the smoke is associated with the first side of thesecond vehicle, step 1930 may determine that the second vehicle emitssmoke not through an exhaust system; in response to a determination thatthe smoke is associated with the second side of the second vehicle, step1930 may withhold and/or forgo determining that the second vehicle emitssmoke not through an exhaust system; and in response to a determinationthat the smoke is not associated with the second vehicle detected bystep 720, step 1930 may withhold and/or forgo determining that thesecond vehicle emits smoke not through an exhaust system.

In some examples, the one or more images obtained by step 710 may beanalyzed to detect smoke. For example, a visual smoke detector may beused to detect smoke in the one or more images obtained by step 710. Inanother example, a machine learning model may be trained using trainingexamples to detect smoke in images and/or videos, and the trainedmachine learning model may be used analyze the one or more imagesobtained by step 710 to detect smoke. An example of such trainingexample may include an image and/or a video, together with a labelindicating whether smoke is depicted in the image and/or in the videoand/or a position of the detected smoke in the image and/or in thevideo. In yet another example, an artificial neural network (such as adeep neural networks, convolutional neural networks, etc.) may beconfigured to detect smoke in images and/or videos, and the artificialneural network may be used analyze the one or more images obtained bystep 710 to detect smoke.

In some examples, the one or more images obtained by step 710 may beanalyzed to determine whether the detected smoke is associated with afirst side of the second vehicle detected by step 720, a second side ofthe second vehicle detected by step 720, or not associated with thesecond vehicle detected by step 720. For example, a position of thedetected smoke may be compared with a position of the different sides ofthe second vehicle detected by step 720, when the detected smoke issubstantially above a side and/or a part of the second vehicle, it maybe determine that the detected smoke is associated with that side and/orwith that part of the second vehicle, and when the detected smoke is notsubstantially above the second vehicle, it may be determined that thedetected smoke is not associated with the second vehicle. In anotherexample, a position of the detected smoke may be compared with aposition of the different sides of the second vehicle detected by step720, when the detected smoke is adjunct to a side and/or part of thesecond vehicle, it may be determine that the detected smoke isassociated with that side and/or that part of the second vehicle, andwhen the detected smoke is not adjunct to the second vehicle, it may bedetermined that the detected smoke is not associated with the secondvehicle. In yet another example, a machine learning model may be trainedusing training examples to associate smoke to parts of vehicles based onimages and/or videos, and the trained machine learning model may be usedto analyze the one or more images obtained by step 710 and associatedthe detected smoke to sides and/or parts of the second vehicle detectedby step 720, or determine that the detected smoke is not associated withthe second vehicle detected by step 720. An example of such trainingexample may include an image depicting a vehicle and smoke, togetherwith a label indicating that the depicted smoke is associated with aparticular side and/or a particular part of the depicted vehicle, ortogether with a label indicating that the depicted smoke is notassociated with the depicted vehicle. In an additional example, anartificial neural network (such as a deep neural networks, convolutionalneural networks, etc.) may be configured to associate smoke to parts ofvehicles based on images and/or videos, and the artificial neuralnetwork may be used to analyze the one or more images obtained by step710 and associated the detected smoke to sides and/or parts of thesecond vehicle detected by step 720.

In some embodiments, the one or more images obtained by step 710 may beanalyzed to determine a color of the smoke emitted not through theexhaust system. Some non-limiting examples of such color may includewhite, black, grey, blue, and so forth. For example, a machine learningmodel may be trained using training examples to determine colors ofsmoke by analyzing images and/or videos, and the trained machinelearning model may be used to analyze the one or more images obtained bystep 710 and determine the color of the smoke emitted not through theexhaust system. An example of such training example may include an imageand/or a video depicting smoke, together with a label indicating thecolor of the depicted smoke. In another example, an artificial neuralnetwork (such as a deep neural networks, convolutional neural networks,etc.) may be configured to determine colors of smoke by analyzing imagesand/or videos, and the artificial neural network may be used to analyzethe one or more images obtained by step 710 and determine the color ofthe smoke emitted not through the exhaust system. In yet anotherexample, a color histogram of an area depicting smoke in the one or moreimages obtained by step 710 may be compared to a color histogram ofother areas (such as nearby areas) of the one or more images obtained bystep 710, and the difference between the two color histograms may beused to identify the color of the smoke emitted not through the exhaustsystem. In some examples, in response to a first determined color of thesmoke (such as white, black, grey, blue, etc.), step 1940 may cause thefirst vehicle to initiate a first action responding to the secondvehicle, and in response to a second determined color of the smoke (suchas white, black, grey, blue, etc.), step 1940 may withhold and/or forgocausing the first vehicle to initiate the first action. In someexamples, in response to a first determined color of the smoke (such aswhite, black, grey, blue, etc.), step 1940 may cause the first vehicleto initiate a first action responding to the second vehicle, and inresponse to a second determined color of the smoke (such as white,black, grey, blue, etc.), step 1940 may cause the first vehicle toinitiate a second action responding to the second vehicle, where thesecond action may differ from the first action.

In some examples, a measurement of a temperature of at least part of thesecond vehicle (such as the engine, the front side, the entire secondvehicle, etc.) may be obtained. For example, the temperature of the atleast part of the second vehicle may be measured using an infraredthermometer. In another example, the measurement of the temperature ofthe at least part of the second vehicle may be received from the secondvehicle (for example, using a point to point communication protocol,through a communication network using a communication device, through acentralized server, and so forth). In yet another example, the one ormore images may comprise at least one infrared image, and the at leastone infrared image may be analyzed to measure the temperature of the atleast part of the second vehicle. For example, a machine learning modelmay be trained using training examples to determine temperatures ofvehicles by analyzing infrared images and/or infrared videos, and thetrained machine learning model may be used to analyze the at least oneinfrared image and determine the measurement of the temperature of theat least part of the second vehicle. An example of such training examplemay include an infrared image and/or an infrared video of a vehicle,together with a label indicating a temperature of the vehicle or of aparticular part of the vehicle. In another example, an artificial neuralnetwork (such as a deep neural networks, convolutional neural networks,etc.) may be configured to determine temperature by analyzing infraredimages and/or infrared videos, and the artificial neural network may beused to analyze the at least one infrared image and determine themeasurement of the temperature of the at least part of the secondvehicle.

In some examples, a type of the second vehicle may be obtained. Forexample, the type of the second vehicle may be received from the secondvehicle (for example, using a point to point communication protocol,through a communication network using a communication device, through acentralized server, and so forth). In another example, a machinelearning model may be trained using training examples to determine typesof vehicles by analyzing images and/or videos, and the trained machinelearning model may be used to analyze the one or more images anddetermine the type of the second vehicle. In yet another example, anartificial neural network (such as a deep neural networks, convolutionalneural networks, etc.) may be configured to determine to determine typesof vehicles by analyzing images and/or videos, and the artificial neuralnetwork may be used to analyze the one or more images and determine thetype of the second vehicle. Further, ins some examples, the type of thesecond vehicle may be used to determine typical temperatures for thesecond vehicle, such as range of temperatures corresponding to a normaloperation of the second vehicle, range of temperatures corresponding toan overheated engine, range of temperatures corresponding to anuncontrolled fire in the second vehicle, range of temperaturescorresponding to range of temperatures corresponding to the secondvehicle being in an off mode, and so forth. For example, a databaseconnecting type of vehicles with ranges of temperatures may be used todetermine typical temperatures for the second vehicle based on the typeof the second vehicle. In another example, a function that determinestypical temperatures for the second vehicle based on properties ofvehicles may be used to determine typical temperatures for the secondvehicle based on properties associated with the type of the secondvehicle.

In some examples, in response to a first temperature of the at leastpart of the second vehicle (such as temperature corresponding to anoverheated engine, temperature corresponding to a fire, temperaturecorresponding to a normal operation temperature of the second vehicle,etc.), step 1940 may cause the first vehicle to initiate a first actionresponding to the second vehicle, and in response to a secondtemperature of the at least part of the second vehicle (such astemperature corresponding to an overheated engine, temperaturecorresponding to a fire, temperature corresponding to a normal operationtemperature of the second vehicle, etc.), step 1940 may withhold and/orforgo causing the first vehicle to initiate the first action. In someexamples, in response to a first temperature of the at least part of thesecond vehicle (such as temperature corresponding to an overheatedengine, temperature corresponding to a fire, temperature correspondingto a normal operation temperature of the second vehicle, etc.), step1940 may cause the first vehicle to initiate a first action respondingto the second vehicle, and in response to a second temperature of the atleast part of the second vehicle (such as temperature corresponding toan overheated engine, temperature corresponding to a fire, temperaturecorresponding to a normal operation temperature of the second vehicle,etc.), step 1940 may cause the first vehicle to initiate a second actionresponding to the second vehicle, where the second action may differfrom the first action. In some examples, in response to a firsttemperature of the at least part of the second vehicle (such astemperature corresponding to an overheated engine, temperaturecorresponding to a fire, temperature corresponding to a normal operationtemperature of the second vehicle, etc.) and a determination by step1930 that the second vehicle emits smoke not through an exhaust system,step 1940 may cause the first vehicle to initiate a first actionresponding to the second vehicle, and in response to a secondtemperature of the at least part of the second vehicle (such astemperature corresponding to an overheated engine, temperaturecorresponding to a fire, temperature corresponding to a normal operationtemperature of the second vehicle, etc.), step 1940 may withhold and/orforgo causing the first vehicle to initiate the first action.

In some examples, a motion of the second vehicle may be determined, forexample as described above. Further, in some examples, in response tothe determined motion of the second vehicle and/or the determinationthat the second vehicle emits smoke not through an exhaust system, thefirst vehicle may be caused to initiate the action (for example asdescribed above). For example, in response to a first determined motionof the second vehicle and the determination that the second vehicleemits smoke not through an exhaust system, the first vehicle may becaused to initiate a first action; and in response to a seconddetermined motion of the second vehicle and the determination that thesecond vehicle emits smoke not through an exhaust system, the firstvehicle may be caused to initiate a second action. In another example,in response to a first determined motion of the second vehicle and thedetermination that the second vehicle emits smoke not through an exhaustsystem, the first vehicle may be caused to initiate a first action; andin response to a second determined motion of the second vehicle and thedetermination that the second vehicle emits smoke not through an exhaustsystem, causing the first vehicle to initiate the first action may bewithheld and/or forwent.

In some examples, it may be determining that the second vehicle issignaling, for example as described above. Further, in some examples, atype of the signaling of the second vehicle may be determined, forexample as described above. Some non-limiting examples of such type ofsignaling may include signaling using light, signaling using sound,signaling left, signaling right, signaling break, signaling hazardwarning, signaling reversing, signaling warning, and so forth. Further,in some examples, in response to the determination that the secondvehicle is signaling and/or the determined type of the signaling and/orthe determination that the second vehicle emits smoke not through anexhaust system, the first vehicle may be caused to initiate the action(for example as described above). For example, in response to a firstdetermined type of the signaling and the determination that the secondvehicle emits smoke not through an exhaust system, the first vehicle maybe caused to initiate a first action; and in response to a seconddetermined type of the signaling and the determination that the secondvehicle emits smoke not through an exhaust system, the first vehicle maybe caused to initiate a second action. In another example, in responseto a first determined type of the signaling and the determination thatthe second vehicle emits smoke not through an exhaust system, the firstvehicle may be caused to initiate a first action; and in response to asecond determined type of the signaling and the determination that thesecond vehicle emits smoke not through an exhaust system, causing thefirst vehicle to initiate the first action may be withheld and/orforwent. In an additional example, in response to the determination thatthe second vehicle is signaling and the determination that the secondvehicle emits smoke not through an exhaust system, the first vehicle maybe caused to initiate a first action; and in response to thedetermination that the second vehicle is not signaling and thedetermination that the second vehicle emits smoke not through an exhaustsystem, the first vehicle may be caused to initiate a second action. Inyet another example, in response to the determination that the secondvehicle is signaling and the determination that the second vehicle emitssmoke not through an exhaust system, the first vehicle may be caused toinitiate a first action; and in response to the determination that thesecond vehicle is not signaling and the determination that the secondvehicle emits smoke not through an exhaust system, causing the firstvehicle to initiate the first action may be withheld and/or forwent.

In some examples, a position of the second vehicle may be determined,for example as described above. For example, the determined position maybe in relation to the ground, in relation to a map, in relation to aroad, in relation to a lane, in relation to an object in theenvironment, in relation to the at least part of the first vehicle, inrelation to the image sensor, and so forth. Further, in some examples,it may be determining that the second vehicle is in a lane of the firstvehicle, for example as described above. Further, in some examples, itmay be determining that the second vehicle is in a planned path of thefirst vehicle, for example as described above. Further, in someexamples, in response to the determined position of the second vehicleand/or the determination that the second vehicle emits smoke not throughan exhaust system, the first vehicle may be caused to initiate theaction (for example as described above). For example, in response to afirst determined position of the second vehicle and the determinationthat the second vehicle emits smoke not through an exhaust system, thefirst vehicle may be caused to initiate a first action; and in responseto a second determined position of the second vehicle and thedetermination that the second vehicle emits smoke not through an exhaustsystem, the first vehicle may be caused to initiate a second action. Inanother example, in response to a first determined position of thesecond vehicle and the determination that the second vehicle emits smokenot through an exhaust system, the first vehicle may be caused toinitiate a first action; and in response to a second determined positionof the second vehicle and the determination that the second vehicleemits smoke not through an exhaust system, causing the first vehicle toinitiate the first action may be withheld and/or forwent. In anadditional example, in response to the determination that the secondvehicle is in a lane and/or a planned path of the first vehicle and thedetermination that the second vehicle emits smoke not through an exhaustsystem, the first vehicle may be caused to initiate a first action; andin response to the determination that the second vehicle is not in alane and/or is not in a planned path of the first vehicle and thedetermination that the second vehicle emits smoke not through an exhaustsystem, the first vehicle may be caused to initiate a second action. Inyet another example, in response to the determination that the secondvehicle is in a lane and/or a planned path of the first vehicle and thedetermination that the second vehicle emits smoke not through an exhaustsystem, the first vehicle may be caused to initiate a first action; andin response to the determination that the second vehicle is not in alane and/or is not in a planned path of the first vehicle and thedetermination that the second vehicle emits smoke not through an exhaustsystem, causing the first vehicle to initiate the first action may bewithheld and/or forwent.

In some embodiments, systems, methods and computer readable media forcontrolling vehicles in response to two-wheeler vehicles are provided.One challenge of autonomous driving is to determine when a standingvehicle has stopped for a short period of time and therefore theautonomous vehicle needs to wait for the standing vehicle to resumemoving, and when the standing vehicle has stopped for a longer period oftime (for example, parking) and therefore the autonomous vehicle needsto pass the standing vehicle. In some cases, identifying that atwo-wheeler vehicle is in use may indicate that the two-wheeler vehiclehas stopped for a longer period of time. The provided systems, methodsand computer readable media for controlling vehicles may detect atwo-wheeler vehicle, determine whether at least one rider rides thetwo-wheeler vehicle, and control a response of an autonomous vehicle tothe detected vehicle based on the determination of whether the at leastone rider rides the two-wheeler vehicle.

FIG. 20 illustrates an example of a method 2000 for controlling vehiclesin response to two-wheeler vehicles. In this example, method 2000 maycomprise: obtaining images captured from an environment of a firstvehicle (Step 710); analyzing the images to detect a two-wheeler vehicle(Step 2020); analyzing the images to determine whether at least onerider rides the two-wheeler vehicle (Step 2030); in response to adetermination that no rider rides the two-wheeler vehicle, causing thefirst vehicle to initiate an action responding to the two-wheelervehicle (Step 2040), and in response to a determination that at leastone rider rides the two-wheeler vehicle, forgoing causing the firstvehicle to initiate the action. In some implementations, method 2000 maycomprise one or more additional steps, while some of the steps listedabove may be modified or excluded. In some implementations, one or moresteps illustrated in FIG. 20 may be executed in a different order and/orone or more groups of steps may be executed simultaneously and/or aplurality of steps may be combined into single step and/or a single stepmay be broken down to a plurality of steps.

In some embodiments, one or more images captured using one or more imagesensors from an environment of a first vehicle may be obtained, forexample as described above. Further, in some examples, the one or moreimages may be analyzed to detect a two-wheeler vehicle. Further, in someexamples, the one or more images may be analyzed to determine whether atleast one rider rides the two-wheeler vehicle. Further, in examples, forexample in response to the determination that no rider rides thetwo-wheeler vehicle, the first vehicle may be caused to initiate anaction responding to the two-wheeler vehicle, for example as describedabove. Some non-limiting examples of such action may include signaling,stopping, changing a speed of the first vehicle, changing a motiondirection of the first vehicle, passing the two-wheeler vehicle,turning, performing a U-turn, driving in reverse, generating an audiblewarning, and so forth.

In some embodiments, step 2020 may comprise analyzing the one or moreimages obtained by step 710 to detect a two-wheeler vehicle (or todetect selected types of two-wheeler vehicles, such as single-tracktwo-wheeler vehicle, a dandy horse, a bicycle, a motorcycle, a dicycle,and so forth). Some non-limiting examples of such two-wheeler vehiclemay include a single-track two-wheeler vehicle, a dandy horse, abicycle, a motorcycle, a dicycle, and so forth. For example, a machinelearning model may be trained using training examples to detecttwo-wheeler vehicle (or to detect selected types of two-wheelervehicles, such as single-track two-wheeler vehicle, a dandy horse, abicycle, a motorcycle, a dicycle, and so forth) in images and/or videos,and step 2020 may use the trained machine learning model to analyze theone or more images obtained by step 710 and detect the two-wheelervehicle in the one or more images. An example of such training examplemay include an image and/or a video, together with a label indicatingwhether a two-wheeler vehicle (or whether a two-wheeler vehicle ofselected types of two-wheeler vehicles, such as single-track two-wheelervehicle, a dandy horse, a bicycle, a motorcycle, a dicycle, and soforth) appears in the image and/or in the video, and/or together with alabel indicating a position of the two-wheeler vehicle in the imageand/or in the video. In another example, an artificial neural network(such as a deep neural networks, convolutional neural networks, etc.)may be configured to detect two-wheeler vehicle (or to detect selectedtypes of two-wheeler vehicles, such as single-track two-wheeler vehicle,a dandy horse, a bicycle, a motorcycle, a dicycle, and so forth) inimages and/or videos, and step 2020 may use the artificial neuralnetwork to analyze the one or more images obtained by step 710 anddetect the two-wheeler vehicle in the one or more images. In yet anotherexample, step 2020 may use object detection algorithms to analyze theone or more images obtained by step 710 and detect the two-wheelervehicle in the one or more images.

In some examples, the one or more images obtained by step 710 may beanalyzed to detect a second vehicle, for example using step 720, andstep 2020 may analyze the one or more images obtained by step 710 todetermine whether the detected second vehicle is a two-wheeler vehicle.For example, a machine learning model may be trained using trainingexamples to determine whether vehicles are two-wheeler vehicles (orwhether vehicles are two-wheeler vehicles of selected types, such assingle-track two-wheeler vehicle, a dandy horse, a bicycle, amotorcycle, a dicycle, etc.) by analyzing images and/or videos, and step2020 may use the trained machine learning model to analyze the one ormore images obtained by step 710 and determine whether the secondvehicle detected by step 720 is a two-wheeler vehicle (or whether thesecond vehicle detected by step 720 is a two-wheeler vehicle of selectedtypes, such as single-track two-wheeler vehicle, a dandy horse, abicycle, a motorcycle, a dicycle, and so forth). An example of suchtraining example may include an image and/or a video depicting avehicle, together with a label indicating whether the depicted vehicleis a two-wheeler vehicle (or whether the depicted vehicle is atwo-wheeler vehicle of selected types, such as single-track two-wheelervehicle, a dandy horse, a bicycle, a motorcycle, a dicycle, and soforth). In another example, an artificial neural network (such as a deepneural networks, convolutional neural networks, etc.) may be configuredto determine whether vehicles are two-wheeler vehicles (or whethervehicles are two-wheeler vehicles of selected types, such assingle-track two-wheeler vehicle, a dandy horse, a bicycle, amotorcycle, a dicycle, etc.) by analyzing images and/or videos, and step2020 may use the artificial neural network to analyze the one or moreimages obtained by step 710 and determine whether the second vehicledetected by step 710 is a two-wheeler vehicle (or whether the secondvehicle detected by step 720 is a two-wheeler vehicle of selected types,such as single-track two-wheeler vehicle, a dandy horse, a bicycle, amotorcycle, a dicycle, and so forth).

In some embodiments, step 2030 may comprise analyzing the one or moreimages obtained by step 710 to determine whether at least one riderrides the two-wheeler vehicle detected by step 2020. For example, amachine learning model may be trained using training examples todetermine whether riders ride two-wheeler vehicles from images and/orvideos, and step 2030 may use the trained machine learning model toanalyze the one or more images obtained by step 710 and determinewhether at least one rider rides the two-wheeler vehicle detected bystep 2020. An example of such training example may include an imageand/or a video depicting a two-wheeler vehicle, together with a labelindicating whether at least one rider rides the depicted two-wheelervehicle. In another example, an artificial neural network (such as adeep neural networks, convolutional neural networks, etc.) may beconfigured to determine whether riders ride two-wheeler vehicles fromimages and/or videos, and step 2030 may use the artificial neuralnetwork to analyze the one or more images obtained by step 710 anddetermine whether at least one rider rides the two-wheeler vehicledetected by step 2020. In yet another example, visual person detectormay be used to detect persons in the one or more images obtained by step710, the location of the detect persons may be compared with thelocation of the two-wheeler vehicle detected by step 2020, and when aperson is detected substantially above the two-wheeler vehicle detectedby step 2020, step 2030 may determine that the detected person is ridingthe two-wheeler vehicle detected by step 2020. Additionally oralternatively, position of two legs of the detected person may beanalyzed to determine whether the two legs of the detected person are onopposite sides of the two-wheeler vehicle detected by step 2020, andwhen the two legs of the detected person are on opposite sides of thetwo-wheeler vehicle detected by step 2020, step 2030 may determine thatthe detected person is riding the two-wheeler vehicle detected by step2020.

In some embodiments, step 2040 may comprise, for example in response toa determination that no rider rides the two-wheeler vehicle detected bystep 2020 (e.g., as determined by step 2030), causing the first vehicleto initiate an action responding to the two-wheeler vehicle. Somenon-limiting examples of such action may include signaling, stopping,changing a speed of the first vehicle, changing a motion direction ofthe first vehicle, passing the two-wheeler vehicle, forgoing passing thetwo-wheeler vehicle, keeping a minimal distance of at least a selectedlength from the two-wheeler vehicle (for example, the selected lengthmay be less than 100 feet, may be at least 100 feet, may be at least 200feet, etc.), turning, performing a U-turn, driving in reverse,generating an audible warning, and so forth. For example, step 2040 maytransmit a signal to an external device (such as a device controllingthe first vehicle, a device navigating the first vehicle, the firstvehicle, a system within the first vehicle, etc.), and the signal may beconfigured to cause the external device to cause the first vehicle toinitiate the action responding to the two-wheeler vehicle detected bystep 2020. In another example, step 2040 may provide information relatedto the two-wheeler vehicle to such external device, and the informationmay be configured to cause the external device to cause the firstvehicle to initiate the action responding to the two-wheeler vehicledetected by step 2020. In one example, in response to a determinationthat at least one rider rides the two-wheeler vehicle detected by step2020 (e.g., as determined by step 2030), step 2040 may cause the firstvehicle to initiate a second action, the second action may differ fromthe action. For example, in response to a determination that no riderrides the two-wheeler vehicle detected by step 2020 (e.g., as determinedby step 2030), step 2040 may cause the first vehicle to initiate a firstaction, and in response to a determination that at least one rider ridesthe two-wheeler vehicle detected by step 2020 (e.g., as determined bystep 2030), step 2040 may cause the first vehicle to initiate a secondaction, the second action may differ from the first action. In anotherexample, in response to a determination that no rider rides thetwo-wheeler vehicle detected by step 2020 (e.g., as determined by step2030), step 2040 may cause the first vehicle to initiate a first action,and in response to a determination that at least one rider rides thetwo-wheeler vehicle detected by step 2020 (e.g., as determined by step2030), step 2040 may withhold and/or forgo causing the first vehicle toinitiate the first action.

In some embodiments, the one or more images obtained by step 710 may beanalyzed to determine whether a leg of the at least one rider of thetwo-wheeler vehicle touches the ground, for example as described below.In some examples, in response to a determination that the leg of the atleast one rider touches the ground, step 2040 may cause the firstvehicle to initiate a first action responding to the two-wheelervehicle, and in response to a determination that the leg of the at leastone rider do not touch the ground, step 2040 may withhold and/or forgocausing the first vehicle to initiate the first action responding to thetwo-wheeler vehicle. In some examples, in response to a determinationthat the leg of the at least one rider touches the ground, step 2040 maycause the first vehicle to initiate a first action responding to thetwo-wheeler vehicle, and in response to a determination that the leg ofthe at least one rider do not touch the ground, step 2040 may cause thefirst vehicle to initiate a second action responding to the two-wheelervehicle, the second action may differ from the first action.

In some examples, the one or more images obtained by step 710 may beanalyzed to determine whether a leg of the at least one rider of thetwo-wheeler vehicle touches the ground. For example, a machine learningmodel may be trained using training examples to determine whether legsof riders of two-wheeler vehicles touch the ground from images and/orvideos, and the trained machine learning model may be used to analyzethe one or more images obtained by step 710 and determine whether a legof the at least one rider of the two-wheeler vehicle touches the ground.An example of such training example may include an image and/or a videodepicting a rider of a two-wheeler vehicle, together with an indicationof whether a leg of the depicted rider touches the ground. In anotherexample, an artificial neural network (such as a deep neural networks,convolutional neural networks, etc.) may be configured to determinewhether legs of riders of two-wheeler vehicles touch the ground fromimages and/or videos, and the artificial neural network may be used toanalyze the one or more images obtained by step 710 and determinewhether a leg of the at least one rider of the two-wheeler vehicletouches ground.

In some examples, a motion of the two-wheeler vehicle may be determined,for example as described above in relation to motion of the secondvehicle. Further, in some examples, in response to the determined motionof the two-wheeler vehicle and/or the determination that no rider ridesthe two-wheeler vehicle, the first vehicle may be caused to initiate theaction (for example as described above). For example, in response to afirst determined motion of the two-wheeler vehicle and the determinationthat no rider rides the two-wheeler vehicle, the first vehicle may becaused to initiate a first action; and in response to a seconddetermined motion of the two-wheeler vehicle and the determination thatno rider rides the two-wheeler vehicle, the first vehicle may be causedto initiate a second action. In another example, in response to a firstdetermined motion of the two-wheeler vehicle and the determination thatno rider rides the two-wheeler vehicle, the first vehicle may be causedto initiate a first action; and in response to a second determinedmotion of the two-wheeler vehicle and the determination that no riderrides the two-wheeler vehicle, causing the first vehicle to initiate thefirst action may be withheld and/or forwent.

In some examples, it may be determining that the two-wheeler vehicle issignaling, for example as described above in relation to signaling ofthe second vehicle. Further, in some examples, a type of the signalingof the two-wheeler vehicle may be determined, for example as describedabove in relation to type of signaling of the second vehicle. Somenon-limiting examples of such type of signaling may include signalingusing light, signaling using sound, signaling left, signaling right,signaling break, signaling hazard warning, signaling reversing,signaling warning, and so forth. Further, in some examples, in responseto the determination that the two-wheeler vehicle is signaling and/orthe determined type of the signaling and/or the determination that norider rides the two-wheeler vehicle, the first vehicle may be caused toinitiate the action (for example as described above). For example, inresponse to a first determined type of the signaling and thedetermination that no rider rides the two-wheeler vehicle, the firstvehicle may be caused to initiate a first action; and in response to asecond determined type of the signaling and the determination that norider rides the two-wheeler vehicle, the first vehicle may be caused toinitiate a second action. In another example, in response to a firstdetermined type of the signaling and the determination that no riderrides the two-wheeler vehicle, the first vehicle may be caused toinitiate a first action; and in response to a second determined type ofthe signaling and the determination that no rider rides the two-wheelervehicle, causing the first vehicle to initiate the first action may bewithheld and/or forwent. In an additional example, in response to thedetermination that the two-wheeler vehicle is signaling and thedetermination that no rider rides the two-wheeler vehicle, the firstvehicle may be caused to initiate a first action; and in response to thedetermination that the two-wheeler vehicle is not signaling and thedetermination that no rider rides the two-wheeler vehicle, the firstvehicle may be caused to initiate a second action. In yet anotherexample, in response to the determination that the two-wheeler vehicleis signaling and the determination that no rider rides the two-wheelervehicle, the first vehicle may be caused to initiate a first action; andin response to the determination that the two-wheeler vehicle is notsignaling and the determination that no rider rides the two-wheelervehicle, causing the first vehicle to initiate the first action may bewithheld and/or forwent.

In some examples, a position of the two-wheeler vehicle may bedetermined, for example as described above in relation to position ofthe second vehicle. For example, the determined position may be inrelation to the ground, in relation to a map, in relation to a road, inrelation to a lane, in relation to an object in the environment, inrelation to the at least part of the first vehicle, in relation to theimage sensor, and so forth. Further, in some examples, it may bedetermining that the two-wheeler vehicle is in a lane of the firstvehicle, for example as described above in relation to the secondvehicle. Further, in some examples, it may be determining that thetwo-wheeler vehicle is in a planned path of the first vehicle, forexample as described above in relation to the second vehicle. Further,in some examples, in response to the determined position of thetwo-wheeler vehicle and/or the determination that no rider rides thetwo-wheeler vehicle, the first vehicle may be caused to initiate theaction (for example as described above). For example, in response to afirst determined position of the two-wheeler vehicle and thedetermination that no rider rides the two-wheeler vehicle, the firstvehicle may be caused to initiate a first action; and in response to asecond determined position of the two-wheeler vehicle and thedetermination that no rider rides the two-wheeler vehicle, the firstvehicle may be caused to initiate a second action. In another example,in response to a first determined position of the two-wheeler vehicleand the determination that no rider rides the two-wheeler vehicle, thefirst vehicle may be caused to initiate a first action; and in responseto a second determined position of the two-wheeler vehicle and thedetermination that no rider rides the two-wheeler vehicle, causing thefirst vehicle to initiate the first action may be withheld and/orforwent. In an additional example, in response to the determination thatthe two-wheeler vehicle is in a lane and/or a planned path of the firstvehicle and the determination that no rider rides the two-wheelervehicle, the first vehicle may be caused to initiate a first action; andin response to the determination that the two-wheeler vehicle is not ina lane and/or is not in a planned path of the first vehicle and thedetermination that no rider rides the two-wheeler vehicle, the firstvehicle may be caused to initiate a second action. In yet anotherexample, in response to the determination that the two-wheeler vehicleis in a lane and/or a planned path of the first vehicle and thedetermination that no rider rides the two-wheeler vehicle, the firstvehicle may be caused to initiate a first action; and in response to thedetermination that the two-wheeler vehicle is not in a lane and/or isnot in a planned path of the first vehicle and the determination that norider rides the two-wheeler vehicle, causing the first vehicle toinitiate the first action may be withheld and/or forwent.

In some examples, a motion of the two-wheeler vehicle may be determined,for example as described above in relation to motion of the secondvehicle. Further, in some examples, in response to the determined motionof the two-wheeler vehicle and/or the determination that at least onerider rides the two-wheeler vehicle, the first vehicle may be caused toinitiate the action (for example as described above). For example, inresponse to a first determined motion of the two-wheeler vehicle and thedetermination that at least one rider rides the two-wheeler vehicle, thefirst vehicle may be caused to initiate a first action; and in responseto a second determined motion of the two-wheeler vehicle and thedetermination that at least one rider rides the two-wheeler vehicle, thefirst vehicle may be caused to initiate a second action. In anotherexample, in response to a first determined motion of the two-wheelervehicle and the determination that at least one rider rides thetwo-wheeler vehicle, the first vehicle may be caused to initiate a firstaction; and in response to a second determined motion of the two-wheelervehicle and the determination that at least one rider rides thetwo-wheeler vehicle, causing the first vehicle to initiate the firstaction may be withheld and/or forwent.

In some examples, it may be determining that the two-wheeler vehicle issignaling, for example as described above in relation to signaling ofthe second vehicle. Further, in some examples, a type of the signalingof the two-wheeler vehicle may be determined, for example as describedabove in relation to type of signaling of the second vehicle. Somenon-limiting examples of such type of signaling may include signalingusing light, signaling using sound, signaling left, signaling right,signaling break, signaling hazard warning, signaling reversing,signaling warning, and so forth. Further, in some examples, in responseto the determination that the two-wheeler vehicle is signaling and/orthe determined type of the signaling and/or the determination that atleast one rider rides the two-wheeler vehicle, the first vehicle may becaused to initiate the action (for example as described above). Forexample, in response to a first determined type of the signaling and thedetermination that at least one rider rides the two-wheeler vehicle, thefirst vehicle may be caused to initiate a first action; and in responseto a second determined type of the signaling and the determination thatat least one rider rides the two-wheeler vehicle, the first vehicle maybe caused to initiate a second action. In another example, in responseto a first determined type of the signaling and the determination thatat least one rider rides the two-wheeler vehicle, the first vehicle maybe caused to initiate a first action; and in response to a seconddetermined type of the signaling and the determination that at least onerider rides the two-wheeler vehicle, causing the first vehicle toinitiate the first action may be withheld and/or forwent. In anadditional example, in response to the determination that thetwo-wheeler vehicle is signaling and the determination that at least onerider rides the two-wheeler vehicle, the first vehicle may be caused toinitiate a first action; and in response to the determination that thetwo-wheeler vehicle is not signaling and the determination that at leastone rider rides the two-wheeler vehicle, the first vehicle may be causedto initiate a second action. In yet another example, in response to thedetermination that the two-wheeler vehicle is signaling and thedetermination that at least one rider rides the two-wheeler vehicle, thefirst vehicle may be caused to initiate a first action; and in responseto the determination that the two-wheeler vehicle is not signaling andthe determination that at least one rider rides the two-wheeler vehicle,causing the first vehicle to initiate the first action may be withheldand/or forwent.

In some examples, a position of the two-wheeler vehicle may bedetermined, for example as described above in relation to position ofthe second vehicle. For example, the determined position may be inrelation to the ground, in relation to a map, in relation to a road, inrelation to a lane, in relation to an object in the environment, inrelation to the at least part of the first vehicle, in relation to theimage sensor, and so forth. Further, in some examples, it may bedetermining that the two-wheeler vehicle is in a lane of the firstvehicle, for example as described above in relation to the secondvehicle. Further, in some examples, it may be determining that thetwo-wheeler vehicle is in a planned path of the first vehicle, forexample as described above in relation to the second vehicle. Further,in some examples, in response to the determined position of thetwo-wheeler vehicle and/or the determination that at least one riderrides the two-wheeler vehicle, the first vehicle may be caused toinitiate the action (for example as described above). For example, inresponse to a first determined position of the two-wheeler vehicle andthe determination that at least one rider rides the two-wheeler vehicle,the first vehicle may be caused to initiate a first action; and inresponse to a second determined position of the two-wheeler vehicle andthe determination that at least one rider rides the two-wheeler vehicle,the first vehicle may be caused to initiate a second action. In anotherexample, in response to a first determined position of the two-wheelervehicle and the determination that at least one rider rides thetwo-wheeler vehicle, the first vehicle may be caused to initiate a firstaction; and in response to a second determined position of thetwo-wheeler vehicle and the determination that at least one rider ridesthe two-wheeler vehicle, causing the first vehicle to initiate the firstaction may be withheld and/or forwent. In an additional example, inresponse to the determination that the two-wheeler vehicle is in a laneand/or a planned path of the first vehicle and the determination that atleast one rider rides the two-wheeler vehicle, the first vehicle may becaused to initiate a first action; and in response to the determinationthat the two-wheeler vehicle is not in a lane and/or is not in a plannedpath of the first vehicle and the determination that at least one riderrides the two-wheeler vehicle, the first vehicle may be caused toinitiate a second action. In yet another example, in response to thedetermination that the two-wheeler vehicle is in a lane and/or a plannedpath of the first vehicle and the determination that at least one riderrides the two-wheeler vehicle, the first vehicle may be caused toinitiate a first action; and in response to the determination that thetwo-wheeler vehicle is not in a lane and/or is not in a planned path ofthe first vehicle and the determination that at least one rider ridesthe two-wheeler vehicle, causing the first vehicle to initiate the firstaction may be withheld and/or forwent.

In some embodiments, it may be determined whether a vehicle (such as thesecond vehicle, the two-wheeler vehicle, etc.) is moving. In oneexample, the a motion of the vehicle may be determined, for example asdescribed above, and the determined motion of the vehicle may beanalyzed to determine whether the vehicle is moving. In another example,the one or more images obtained by step 710 may be analyzed to determinewhether the vehicle is moving. In yet another example, a machinelearning model may be trained using training examples to determinewhether vehicles are moving from images and/or videos, and the trainedmachine learning model may be used to analyze the one or more imagesobtained by step 710 and determine whether the vehicle is moving. Anexample of such training example may include an image and/or a videodepicting a vehicle, together with a label indicating whether thedepicted vehicle is moving. In one example, in response to adetermination that the vehicle (such as the second vehicle, thetwo-wheeler vehicle, etc.) is not moving, an action (such as causing thefirst vehicle to perform an action as described herein) may beperformed, and in response to a determination that the vehicle (such asthe second vehicle, the two-wheeler vehicle, etc.) is moving, performingthe action may be forgone. In another example, in response to adetermination that the vehicle (such as the second vehicle, thetwo-wheeler vehicle, etc.) is moving, an action (such as causing thefirst vehicle to perform an action as described herein) may beperformed, and in response to a determination that the vehicle (such asthe second vehicle, the two-wheeler vehicle, etc.) is not moving,performing the action may be forgone. In one example, in response to adetermination that the two-wheeler vehicle is not moving and thedetermination by step 2030 that no rider rides the two-wheeler vehicle,step 2040 may cause the first vehicle to initiate the action. In oneexample, in response to the determination that the two-wheeler vehicleis not moving and a determination by step 2030 that at least one riderrides the two-wheeler vehicle, step 2040 may forgo causing the firstvehicle to initiate the action. In one example, in response to thedetermination that the two-wheeler vehicle is moving, step 2040 mayforgo causing the first vehicle to initiate the action.

In some embodiments, it may be determined whether a vehicle (such as thesecond vehicle, the two-wheeler vehicle, etc.) is an autonomous vehicleconfigured to drive without a driver (such as an autonomous two-wheelervehicle configured to drive without a driver). In one example, the oneor more images obtained by step 710 may be analyzed to determine whetherthe vehicle is an autonomous vehicle configured to drive without adriver. In another example, a machine learning model may be trainedusing training examples to determine whether vehicles are autonomousvehicles configured to drive without drivers from images and/or videos,and the trained machine learning model may be used to analyze the one ormore images obtained by step 710 and determine whether the vehicle is anautonomous vehicle configured to drive without a driver. An example ofsuch training example may include an image and/or a video depicting avehicle, together with a label indicating whether the depicted vehicleis an autonomous vehicle configured to drive without a driver. In yetanother example, a model of the vehicle may be identified (for exampleusing object recognition algorithms), and the model may be used todetermine whether the vehicle is an autonomous vehicle configured todrive without a driver. In an additional example, a visual indicatorvisible on the vehicle may be detected (for example using objectdetection algorithms), and the detected visual indicator may be used todetermine whether the vehicle is an autonomous vehicle configured todrive without a driver. In some examples, in response to a determinationthat the vehicle is an autonomous vehicle configured to drive without adriver, an action (such as causing the first vehicle to initiate theaction) may be performed, and in response to a determination that thevehicle is not an autonomous vehicle configured to drive without adriver, performing the action may be forgone and/or a different actionmay be performed. In some examples, in response to a determination thatthe vehicle is not an autonomous vehicle configured to drive without adriver, an action (such as causing the first vehicle to initiate theaction) may be performed, and in response to a determination that thevehicle is an autonomous vehicle configured to drive without a driver,performing the action may be forgone. In some examples, in response to adetermination by step 2030 that the two-wheeler vehicle is an autonomoustwo-wheeler vehicle configured to drive without a driver and thedetermination that no rider rides the two-wheeler vehicle, step 2040 mayperform an action (such as causing the first vehicle to initiate theaction), and in response to a determination that the two-wheeler vehicleis not an autonomous two-wheeler vehicle configured to drive without adriver and the determination by step 2030 that no rider rides thetwo-wheeler vehicle, step 2040 may forgo performing the action. In someexamples, in response to a determination that the two-wheeler vehicle isnot an autonomous two-wheeler vehicle configured to drive without adriver and the determination by step 2030 that no rider rides thetwo-wheeler vehicle, step 2040 may perform an action (such as causingthe first vehicle to initiate the action), and in response to adetermination that the two-wheeler vehicle is an autonomous two-wheelervehicle configured to drive without a driver and the determination bystep 2030 that no rider rides the two-wheeler vehicle, step 2040 mayforgo performing the action. In some examples, in response to adetermination that the two-wheeler vehicle is not an autonomoustwo-wheeler vehicle configured to drive without a driver and thedetermination by step 2030 that no rider rides the two-wheeler vehicle,step 2040 may perform a first action, and in response to a determinationthat the two-wheeler vehicle is an autonomous two-wheeler vehicleconfigured to drive without a driver and the determination by step 2030that no rider rides the two-wheeler vehicle, step 2040 may perform the asecond action, the second action may differ from the first action.

In some embodiments, systems, methods and computer readable media forcontrolling vehicles in response to winter service vehicles areprovided. In some embodiments, systems, methods and computer readablemedia for controlling vehicles in response to winter service vehiclesare provided. One challenge of autonomous driving is to determine when avehicle is moving slowly for a short period of time and therefore theautonomous vehicle needs to wait for the slow vehicle to resume normalspeed, and when the vehicle is moving slowly for a longer period of timeand the autonomous vehicle needs to pass the slow moving vehicle. Insome cases, identifying that the vehicle is a winter service vehicle mayindicate that the vehicle is moving slowly for a longer period of time.The provided systems, methods and computer readable media forcontrolling vehicles may detect a vehicle, identify whether the vehicleis a winter service vehicle, and control a response of an autonomousvehicle to the detected vehicle based on whether the vehicle is a winterservice vehicle.

FIG. 21 illustrates an example of a method 2100 for controlling vehiclesin response to winter service vehicles. In this example, method 2100 maycomprise: obtaining images captured from an environment of a firstvehicle (Step 710); analyzing the images to detect a second vehicle(Step 720); analyzing the images to determine that the second vehicle isa winter service vehicle (Step 2130); in response to the determinationthat the second vehicle is a winter service vehicle, causing the firstvehicle to initiate an action responding to the second vehicle (Step2140), and in response to the determination that the second vehicle isnot a winter service vehicle, forgoing causing the first vehicle toinitiate the action. In some implementations, method 2100 may compriseone or more additional steps, while some of the steps listed above maybe modified or excluded. In some implementations, one or more stepsillustrated in FIG. 21 may be executed in a different order and/or oneor more groups of steps may be executed simultaneously and/or aplurality of steps may be combined into single step and/or a single stepmay be broken down to a plurality of steps.

In some embodiments, one or more images captured using one or more imagesensors from an environment of a first vehicle may be obtained, forexample as described above. Further, in some examples, the one or moreimages may be analyzed to detect a second vehicle, for example asdescribed above. Further, in some examples, the one or more images maybe analyzed to determine whether the second vehicle is a winter servicevehicle. Further, in examples, for example in response to thedetermination that the second vehicle is a winter service vehicle, thefirst vehicle may be caused to initiate an action responding to thesecond vehicle, for example as described above. Some non-limitingexamples of such action may include signaling, stopping, changing aspeed of the first vehicle, changing a motion direction of the firstvehicle, passing the second vehicle, forgoing passing the secondvehicle, turning, performing a U-turn, driving in reverse, generating anaudible warning, keeping a minimal distance of at least a selectedlength from the second vehicle (for example, where the selected distanceis at least 20 feet, at least 50 feet, at least 100 feet, at least 190feet, at least 200 feet, etc.), and so forth.

In some embodiments, step 2130 may comprise analyzing the one or moreimages obtained by step 710 to determine whether the second vehicledetected by step 720 is a winter service vehicle. For example, a machinelearning model may be trained using training examples to determinewhether vehicles are winter service vehicles by analyzing images and/orvideos, and step 2130 may use the trained machine learning model toanalyze the one or more images obtained by step 710 and determinewhether the second vehicle detected by step 720 is a winter servicevehicle. An example of such training example may include an image and/ora video depicting a vehicle, together with a label indicating whetherthe depicted vehicle is a winter service vehicle. In another example, anartificial neural network (such as a deep neural networks, convolutionalneural networks, etc.) may be configured to determine whether vehiclesare winter service vehicles by analyzing images and/or videos, and step2130 may use the artificial neural network to analyze the one or moreimages obtained by step 710 and determine whether the second vehicledetected by step 720 is a winter service vehicle. In yet anotherexample, step 2130 may use image classification algorithms to analyzethe one or more images obtained by step 710 and determine whether thesecond vehicle detected by step 720 is a winter service vehicle.

In some embodiments, step 2140 may comprise, for example in response toa determination that the second vehicle detected by step 720 is a winterservice vehicle (e.g., as determined by step 2130), causing the firstvehicle to initiate an action responding to the second vehicle detectedby step 720. Some non-limiting examples of such action may includesignaling, stopping, changing a speed of the first vehicle, changing amotion direction of the first vehicle, passing the second vehicle,forgoing passing the second vehicle, keeping a minimal distance of atleast a selected length from the second vehicle (for example, theselected length may be less than 100 feet, may be at least 100 feet, maybe at least 200 feet, etc.), turning, performing a U-turn, driving inreverse, generating an audible warning, and so forth. For example, step2140 may transmit a signal to an external device (such as a devicecontrolling the first vehicle, a device navigating the first vehicle,the first vehicle, a system within the first vehicle, etc.), and thesignal may be configured to cause the external device to cause the firstvehicle to initiate the action responding to the second vehicle detectedby step 720. In another example, step 2140 may provide informationrelated to the second vehicle detected by step 720 (such as position,motion, type, activity status, dimensions, etc.) to such externaldevice, and the information may be configured to cause the externaldevice to cause the first vehicle to initiate the action responding tothe second vehicle detected by step 720. In one example, in response toa determination that the second vehicle detected by step 720 is not awinter service vehicle (e.g., as determined by step 2130), step 2140 maycause the first vehicle to initiate a second action, the second actionmay differ from the action. In one example, in response to adetermination that the second vehicle detected by step 720 is a winterservice vehicle (e.g., as determined by step 2130), step 2140 may causethe first vehicle to initiate a first action, and in response to adetermination that the second vehicle detected by step 720 is not awinter service vehicle (e.g., as determined by step 2130), step 2140 maycause the first vehicle to initiate a second action, the second actionmay differ from the first action. In another example, in response to adetermination that the second vehicle detected by step 720 is a winterservice vehicle (e.g., as determined by step 2130), step 2140 may causethe first vehicle to initiate a first action, and in response to adetermination that the second vehicle detected by step 720 is not awinter service vehicle (e.g., as determined by step 2130), step 2140 maywithhold and/or forgo causing the first vehicle to initiate the firstaction.

In some embodiments, the one or more images obtained by step 710 may beanalyzed to determine whether a snow plough is mounted to the secondvehicle detected by step 720, for example as described below. Further,in some examples, in response to the determination that a snow plough ismounted to the second vehicle detected by step 720, step 2130 maydetermine that the second vehicle is a winter service vehicle. Further,in some examples, in response to the determination that no snow ploughis mounted to the second vehicle detected by step 720, step 2130 maydetermine that the second vehicle is not a winter service vehicle.

In some examples, the one or more images obtained by step 710 may beanalyzed to determine whether a snow plough is mounted to the secondvehicle detected by step 720. For example, a machine learning model maybe trained using training examples to determine whether vehicles aremounted to snow ploughs, and the trained machine learning model may beused to analyze the one or more images obtained by step 710 anddetermine whether a snow plough is mounted to the second vehicledetected by step 720. An example of such training example may include animage and/or a video depicting a vehicle, together with a labelindicating whether the depicted vehicle is connected to a snow plough.In another example, an artificial neural network (such as a deep neuralnetworks, convolutional neural networks, etc.) may be configured todetermine whether vehicles are mounted to snow ploughs from imagesand/or videos, and the artificial neural network may be used to analyzethe one or more images and determine whether a snow plough is mounted tothe second vehicle. In yet another example, an object detectionalgorithms (such as a visual snow plough detector) may be used to detectsnow ploughs in the one or more images obtained by step 710, thelocation of the detected snow ploughs may be compared to the location ofthe second vehicle detected by step 720, and when a location of a snowplough is adjunct to the location of the second vehicle it may bedetermined that the snow plough is mounted to the second vehicle.

In some embodiments, the one or more images obtained by step 710 may beanalyzed to detect snow moving in a vicinity of the second vehicledetected by step 720, for example as described below. Further, in someexamples, the one or more images obtained by step 710 to identify apattern of motion of the snow moving in the vicinity of the secondvehicle detected by step 720, for example as described below. In oneexample, the identified pattern of the motion of the snow moving in thevicinity of the second vehicle detected by step 720 may be analyzed todetermine whether the second vehicle detected by step 720 is a winterservice vehicle, for example as described below. In one example, theidentified pattern of the motion of the snow moving in the vicinity ofthe second vehicle detected by step 720 may be analyzed to determine astate of the winter service vehicle. Some non-limiting examples of suchstate may include engaged in removal of at least one of snow and ice,engaged in removal of snow, engaged in removal of ice, not engaged inremoval of snow, not engaged in removal of ice, not engaged in removalof any one of snow and ice, and so forth.

In some examples, the one or more images obtained by step 710 may beanalyzed to detect snow moving in a vicinity of the second vehicle. Forexample, a machine learning model may be trained using training examplesto detect moving snow in images and/or videos, and the trained machinelearning model may be used to analyze the one or more images obtained bystep 710 and detect snow moving in a vicinity of the second vehicledetected by step 720. An example of such training example may include animage and/or a video, together with a label indicating whether snow ismoving in particular areas of the image and/or the video. In anotherexample, an artificial neural network (such as a deep neural networks,convolutional neural networks, etc.) may be configured to detect movingsnow in images and/or videos, and the artificial neural network may beused to analyze the one or more images obtained by step 710 and detectsnow moving in a vicinity of the second vehicle detected by step 720. Inyet another example, a motion object detection algorithm may be used todetect snow moving in a vicinity of the second vehicle detected by step720 in the one or more images obtained by step 710.

In some examples, the one or more images obtained by step 710 toidentify a pattern of motion of the snow moving in the vicinity of thesecond vehicle detected by step 720. Some non-limiting examples of suchpatterns of motion of snow may include an average direction of motion,an average speed of motion, an histogram of directions of motion, anhistogram of speeds of motion, a mapping of an average direction ofmotion at different areas, a mapping of an average speed of motion atdifferent areas, a mapping of histograms of directions of motion atdifferent areas, a mapping of histograms of speeds of motion atdifferent areas, any combination of the above, a qualitative descriptionof the motion (such as ‘flat’, ‘solid stream’, ‘full cone’, ‘hollowcone’, ‘dropping’, ‘spraying’, etc.), and so forth. For example, amachine learning model may be trained using training examples toidentify patterns of motion of moving snow in images and/or videos, andthe trained machine learning model may be used to analyze the one ormore images obtained by step 710 and identify the pattern of motion ofthe snow moving in the vicinity of the second vehicle detected by step720. An example of such training example may include an image and/or avideo of a moving snow, together with a label indicating the pattern ofmotion of the snow. In another example, an artificial neural network(such as a deep neural networks, convolutional neural networks, etc.)may be configured to identify patterns of motion of moving snow inimages and/or videos, and the artificial neural network may be used toanalyze the one or more images obtained by step 710 and identify thepattern of motion of the snow moving in the vicinity of the secondvehicle detected by step 720. In yet another example, a motionestimation algorithm may be used to identify the pattern of motion ofthe snow moving in the vicinity of the second vehicle.

In some examples, the identified pattern of the motion of the snowmoving in the vicinity of the second vehicle detected by step 720 may beanalyzed to determine whether the second vehicle detected by step 720 isa winter service vehicle. For example, a machine learning model may betrained using training examples to determine whether vehicles are winterservice vehicles based on patterns of motion of snow moving in thevicinity of the vehicles, and the trained machine learning model may beused to analyze the identified pattern of the motion of the snow movingin the vicinity of the second vehicle detected by step 720 and determinewhether the second vehicle detected by step 720 is a winter servicevehicle. An example of such training example may include a pattern ofmotion of snow in a vicinity of a vehicle, together with a labelindicating whether the vehicle is a winter service vehicle. In anotherexample, an artificial neural network may be configured to determinewhether vehicles are winter service vehicles based on patterns of motionof snow moving in the vicinity of the vehicles, and the artificialneural network may be used to analyze the identified pattern of themotion of the snow moving in the vicinity of the second vehicle detectedby step 720 and determine whether the second vehicle is a winter servicevehicle detected by step 720.

In some examples, the identified pattern of the motion of the snowmoving in the vicinity of the second vehicle detected by step 720 may beanalyzed to determine a state of the winter service vehicle. Forexample, a machine learning model may be trained using training examplesto determine states of winter service vehicles based on patterns ofmotion of snow moving in the vicinity of the winter service vehicles,and the trained machine learning model may be used to analyze theidentified pattern of the motion of the snow moving in the vicinity ofthe second vehicle detected by step 720 and determine the state of thewinter service vehicle. An example of such training example may includea pattern of motion of snow, together with a label indicating the stateof the winter service vehicle to be identified. In another example, anartificial neural network may be configured to determine states ofwinter service vehicles based on patterns of motion of snow moving inthe vicinity of the winter service vehicles, and the artificial neuralnetwork may be used to analyze the identified pattern of the motion ofthe snow moving in the vicinity of the second vehicle and determine thestate of the winter service vehicle.

In some embodiments, a state of the winter service vehicle may bedetermined, for example as described below. Some non-limiting examplesof such state may include engaged in removal of at least one of snow andice, engaged in removal of snow, engaged in removal of ice, not engagedin removal of snow, not engaged in removal of ice, not engaged inremoval of any one of snow and ice, and so forth. In some examples, inresponse to a first determined state of the winter service vehicle (suchas engaged in removal of at least one of snow and ice, engaged inremoval of snow, engaged in removal of ice, not engaged in removal ofsnow, not engaged in removal of ice, not engaged in removal of any oneof snow and ice, etc.), step 2140 may cause the first vehicle toinitiate a first action responding to the second vehicle, and inresponse to a second determined state of the winter service vehicle(such as engaged in removal of at least one of snow and ice, engaged inremoval of snow, engaged in removal of ice, not engaged in removal ofsnow, not engaged in removal of ice, not engaged in removal of any oneof snow and ice, etc.), step 2140 may withhold and/or forgo causing thefirst vehicle to initiate the first action. In some examples, inresponse to a first determined state of the winter service vehicle (suchas engaged in removal of at least one of snow and ice, engaged inremoval of snow, engaged in removal of ice, not engaged in removal ofsnow, not engaged in removal of ice, not engaged in removal of any oneof snow and ice, etc.), step 2140 may cause the first vehicle toinitiate a first action responding to the second vehicle, and inresponse to a second determined state of the winter service vehicle(such as engaged in removal of at least one of snow and ice, engaged inremoval of snow, engaged in removal of ice, not engaged in removal ofsnow, not engaged in removal of ice, not engaged in removal of any oneof snow and ice, etc.), step 2140 may cause the first vehicle toinitiate a second action responding to the second vehicle, where thesecond action may differ from the first action.

In some examples, the one or more images obtained by step 710 may beanalyzed to determine a state of the winter service vehicle. Forexample, the one or more images obtained by step 710 may be analyzed todetermine a state of the winter service vehicle based on patterns ofmotion of snow as described above. In another example, a machinelearning model may be trained using training examples to determinestates of winter service vehicles by analyzing images and/or videos, andthe trained machine learning model may be used to analyze the one ormore images obtained by step 710 and determine the state of the winterservice vehicle. An example of such training example may include animage and/or a video depicting a winter service vehicle, together with alabel indicating the state of the depicted winter service vehicle. Inanother example, an artificial neural network (such as a deep neuralnetworks, convolutional neural networks, etc.) may be configured todetermine states of winter service vehicles by analyzing images and/orvideos, and the artificial neural network may be used to analyze the oneor more images obtained by step 710 and determine the state of thewinter service vehicle.

In some examples, audio data captured using one or more audio sensorsfrom an environment of the first vehicle may be obtained and/or analyzedto determine a state of the winter service vehicle. For example, amachine learning model may be trained using training examples todetermine states of winter service vehicles by analyzing audio input,and the trained machine learning model may be used to analyze thecaptured audio data and determine the state of the winter servicevehicle. An example of such training example may include an audiorecording, together with a label indicating the state of the depictedwinter service vehicle. In another example, an artificial neural network(such as a recurrent neural networks, long short-term memory neuralnetworks, etc.) may be configured to determine states of winter servicevehicles by analyzing audio input, and the artificial neural network maybe used to analyze the captured audio data and determine the state ofthe winter service vehicle. In yet another example, signal processingalgorithms may be used to analyze the captured audio data and determinewhether or not the winter service vehicle is engaged in removal of snowand/or ice.

In some examples, a motion of the second vehicle may be determined, forexample as described above. Further, in some examples, in response tothe determined motion of the second vehicle and/or the determinationthat the second vehicle is a winter service vehicle and/or based on thedetermined state of the winter service vehicle, the first vehicle may becaused to initiate the action (for example as described above). Forexample, in response to a first determined motion of the second vehicleand the determination that the second vehicle is a winter servicevehicle, the first vehicle may be caused to initiate a first action; inresponse to a second determined motion of the second vehicle and thedetermination that the second vehicle is a winter service vehicle, thefirst vehicle may be caused to initiate a second action; and in responseto the first determined motion of the second vehicle and thedetermination that the second vehicle is not a winter service vehicle,the first vehicle may be caused to initiate a third action. In anotherexample, in response to a first determined motion of the second vehicleand the determination that the second vehicle is a winter servicevehicle, the first vehicle may be caused to initiate a first action; andin response to a second determined motion of the second vehicle and thedetermination that the second vehicle is a winter service vehicle,causing the first vehicle to initiate the first action may be withheldand/or forwent. For example, in response to a first determined motion ofthe second vehicle and a first determined state of the winter servicevehicle, the first vehicle may be caused to initiate a first action; inresponse to a second determined motion of the second vehicle and thefirst determined state of the winter service vehicle, the first vehiclemay be caused to initiate a second action; and in response to the firstdetermined motion of the second vehicle and a second determined state ofthe winter service vehicle, the first vehicle may be caused to initiatea third action. In another example, in response to a first determinedmotion of the second vehicle and a first determined state of the winterservice vehicle, the first vehicle may be caused to initiate a firstaction; in response to a second determined motion of the second vehicleand the first determined state of the winter service vehicle and/or inresponse to the first determined motion of the second vehicle and asecond determined state of the winter service vehicle, causing the firstvehicle to initiate the first action may be withheld and/or forwent.

In some examples, it may be determining that the second vehicle issignaling, for example as described above. Further, in some examples, atype of the signaling of the second vehicle may be determined, forexample as described above. Some non-limiting examples of such type ofsignaling may include signaling using light, signaling using sound,signaling left, signaling right, signaling break, signaling hazardwarning, signaling reversing, signaling warning, and so forth. Further,in some examples, in response to the determination that the secondvehicle is signaling and/or the determined type of the signaling and/orthe determination that the second vehicle is a winter service vehicleand/or based on the determined state of the winter service vehicle, thefirst vehicle may be caused to initiate the action (for example asdescribed above). For example, in response to a first determined type ofthe signaling and the determination that the second vehicle is a winterservice vehicle, the first vehicle may be caused to initiate a firstaction; in response to a second determined type of the signaling and thedetermination that the second vehicle is a winter service vehicle, thefirst vehicle may be caused to initiate a second action; and in responseto the first determined type of the signaling and the determination thatthe second vehicle is not a winter service vehicle, the first vehiclemay be caused to initiate a third action. In another example, inresponse to a first determined type of the signaling and thedetermination that the second vehicle is a winter service vehicle, thefirst vehicle may be caused to initiate a first action; in response to asecond determined type of the signaling and the determination that thesecond vehicle is a winter service vehicle and/or in response to thefirst determined type of the signaling and the determination that thesecond vehicle is not a winter service vehicle, causing the firstvehicle to initiate the first action may be withheld and/or forwent. Forexample, in response to a first determined type of the signaling and afirst determined state of the winter service vehicle, the first vehiclemay be caused to initiate a first action; in response to a seconddetermined type of the signaling and the first determined state of thewinter service vehicle, the first vehicle may be caused to initiate asecond action; and in response to the first determined type of thesignaling and a second determined state of the winter service vehicle,the first vehicle may be caused to initiate a third action. In anotherexample, in response to a first determined type of the signaling and afirst determined state of the winter service vehicle, the first vehiclemay be caused to initiate a first action; in response to a seconddetermined type of the signaling and the first determined state of thewinter service vehicle and/or in response to the first determined typeof the signaling and a second determined state of the winter servicevehicle, causing the first vehicle to initiate the first action may bewithheld and/or forwent. In an additional example, in response to thedetermination that the second vehicle is signaling and the determinationthat the second vehicle is a winter service vehicle, the first vehiclemay be caused to initiate a first action; and in response to thedetermination that the second vehicle is not signaling and thedetermination that the second vehicle is a winter service vehicle, thefirst vehicle may be caused to initiate a second action. In yet anotherexample, in response to the determination that the second vehicle issignaling and the determination that the second vehicle is a winterservice vehicle, the first vehicle may be caused to initiate a firstaction; and in response to the determination that the second vehicle isnot signaling and the determination that the second vehicle is a winterservice vehicle, causing the first vehicle to initiate the first actionmay be withheld and/or forwent. In an additional example, in response tothe determination that the second vehicle is signaling and a firstdetermined state of the winter service vehicle, the first vehicle may becaused to initiate a first action; and in response to the determinationthat the second vehicle is not signaling and the first determined stateof the winter service vehicle, the first vehicle may be caused toinitiate a second action. In yet another example, in response to thedetermination that the second vehicle is signaling and a firstdetermined state of the winter service vehicle, the first vehicle may becaused to initiate a first action; and in response to the determinationthat the second vehicle is not signaling and the first determined stateof the winter service vehicle, causing the first vehicle to initiate thefirst action may be withheld and/or forwent.

In some examples, a position of the second vehicle may be determined,for example as described above. For example, the determined position maybe in relation to the ground, in relation to a map, in relation to aroad, in relation to a lane, in relation to an object in theenvironment, in relation to the at least part of the first vehicle, inrelation to the image sensor, and so forth. Further, in some examples,it may be determining that the second vehicle is in a lane of the firstvehicle, for example as described above. Further, in some examples, itmay be determining that the second vehicle is in a planned path of thefirst vehicle, for example as described above. Further, in someexamples, in response to the determined position of the second vehicleand/or the determination that the second vehicle is a winter servicevehicle and/or based the first determined state of the winter servicevehicle, the first vehicle may be caused to initiate the action (forexample as described above). For example, in response to a firstdetermined position of the second vehicle and the determination that thesecond vehicle is a winter service vehicle, the first vehicle may becaused to initiate a first action; in response to a second determinedposition of the second vehicle and the determination that the secondvehicle is a winter service vehicle, the first vehicle may be caused toinitiate a second action; and in response to the first determinedposition of the second vehicle and the determination that the secondvehicle is not a winter service vehicle, the first vehicle may be causedto initiate a third action. In another example, in response to a firstdetermined position of the second vehicle and the determination that thesecond vehicle is a winter service vehicle, the first vehicle may becaused to initiate a first action; in response to a second determinedposition of the second vehicle and the determination that the secondvehicle is a winter service vehicle and/or in response to the firstdetermined position of the second vehicle and the determination that thesecond vehicle is not a winter service vehicle, causing the firstvehicle to initiate the first action may be withheld and/or forwent. Inan additional example, in response to the determination that the secondvehicle is in a lane and/or a planned path of the first vehicle and thedetermination that the second vehicle is a winter service vehicle, thefirst vehicle may be caused to initiate a first action; and in responseto the determination that the second vehicle is not in a lane and/or isnot in a planned path of the first vehicle and the determination thatthe second vehicle is a winter service vehicle, the first vehicle may becaused to initiate a second action. In yet another example, in responseto the determination that the second vehicle is in a lane and/or aplanned path of the first vehicle and the determination that the secondvehicle is a winter service vehicle, the first vehicle may be caused toinitiate a first action; and in response to the determination that thesecond vehicle is not in a lane and/or is not in a planned path of thefirst vehicle and the determination that the second vehicle is a winterservice vehicle, causing the first vehicle to initiate the first actionmay be withheld and/or forwent. For example, in response to a firstdetermined position of the second vehicle and a first determined stateof the winter service vehicle, the first vehicle may be caused toinitiate a first action; in response to a second determined position ofthe second vehicle and the first determined state of the winter servicevehicle, the first vehicle may be caused to initiate a second action;and in response to the first determined position of the second vehicleand a second determined state of the winter service vehicle, the firstvehicle may be caused to initiate a third action. In another example, inresponse to a first determined position of the second vehicle and afirst determined state of the winter service vehicle, the first vehiclemay be caused to initiate a first action; in response to a seconddetermined position of the second vehicle and the first determined stateof the winter service vehicle and/or in response to the first determinedposition of the second vehicle and a second determined state of thewinter service vehicle, causing the first vehicle to initiate the firstaction may be withheld and/or forwent. In an additional example, inresponse to the determination that the second vehicle is in a laneand/or a planned path of the first vehicle and a first determined stateof the winter service vehicle, the first vehicle may be caused toinitiate a first action; and in response to the determination that thesecond vehicle is not in a lane and/or is not in a planned path of thefirst vehicle and the first determined state of the winter servicevehicle, the first vehicle may be caused to initiate a second action. Inyet another example, in response to the determination that the secondvehicle is in a lane and/or a planned path of the first vehicle and afirst determined state of the winter service vehicle, the first vehiclemay be caused to initiate a first action; and in response to thedetermination that the second vehicle is not in a lane and/or is not ina planned path of the first vehicle and the first determined state ofthe winter service vehicle, causing the first vehicle to initiate thefirst action may be withheld and/or forwent.

In some embodiments, systems, methods and computer readable media forcontrolling vehicles in response to waste collection vehicles areprovided. In some embodiments, systems, methods and computer readablemedia for controlling vehicles in response to waste collection vehiclesare provided. One challenge of autonomous driving is to determine when avehicle is moving slowly for a short period of time and therefore theautonomous vehicle needs to wait for the slow vehicle to resume normalspeed, and when the vehicle is moving slowly for a longer period of timeand the autonomous vehicle needs to pass the slow moving vehicle. Insome cases, identifying that the vehicle is a waste collection vehiclemay indicate that the vehicle is moving slowly for a longer period oftime. The provided systems, methods and computer readable media forcontrolling vehicles may detect a vehicle, identify whether the vehicleis a waste collection vehicle, and control a response of an autonomousvehicle to the detected vehicle based on whether the vehicle is a wastecollection vehicle.

FIG. 22 illustrates an example of a method 2200 for controlling vehiclesin response to waste collection vehicles. In this example, method 2200may comprise: obtaining images captured from an environment of a firstvehicle (Step 710); analyzing the images to detect a second vehicle(Step 720); analyzing the images to determine that the second vehicle isa waste collection vehicle (Step 2230); in response to the determinationthat the second vehicle is a waste collection vehicle, causing the firstvehicle to initiate an action responding to the second vehicle (Step2240), and in response to the determination that the second vehicle isnot a waste collection vehicle, forgoing causing the first vehicle toinitiate the action. In some implementations, method 2200 may compriseone or more additional steps, while some of the steps listed above maybe modified or excluded. In some implementations, one or more stepsillustrated in FIG. 22 may be executed in a different order and/or oneor more groups of steps may be executed simultaneously and/or aplurality of steps may be combined into single step and/or a single stepmay be broken down to a plurality of steps.

In some embodiments, one or more images captured using one or more imagesensors from an environment of a first vehicle may be obtained, forexample as described above. Further, in some examples, the one or moreimages may be analyzed to detect a second vehicle, for example asdescribed above. Further, in some examples, the one or more images maybe analyzed to determine whether the second vehicle is a wastecollection vehicle. Further, in examples, for example in response to thedetermination that the second vehicle is a waste collection vehicle, thefirst vehicle may be caused to initiate an action responding to thesecond vehicle, for example as described above. Some non-limitingexamples of such action may include signaling, stopping, changing aspeed of the first vehicle, changing a motion direction of the firstvehicle, passing the second vehicle, forgoing passing the secondvehicle, keeping a minimal distance of at least a selected length fromthe second vehicle (for example, where the selected distance is at least20 feet, at least 50 feet, at least 100 feet, at least 190 feet, atleast 200 feet, etc.), turning, performing a U-turn, driving in reverse,generating an audible warning, and so forth.

In some embodiments, step 2230 may comprise analyzing the one or moreimages obtained by step 710 to determine whether the second vehicledetected by step 720 is a waste collection vehicle. Some non-limitingexamples of such state may include engaged in waste collection, notengaged in waste collection, and so forth. For example, a machinelearning model may be trained using training examples to determinewhether vehicles are waste collection vehicles by analyzing imagesand/or videos, and step 2230 may use the trained machine learning modelto analyze the one or more images obtained by step 710 and determinewhether the second vehicle detected by step 720 is a waste collectionvehicle. An example of such training example may include an image and/ora video depicting a vehicle, together with a label indicating whetherthe depicted vehicle is a waste collection vehicle. In another example,an artificial neural network (such as a deep neural networks,convolutional neural networks, etc.) may be configured to determinewhether vehicles are waste collection vehicles by analyzing imagesand/or videos, and step 2230 may use the artificial neural network toanalyze the one or more images obtained by step 710 and determinewhether the second vehicle detected by step 720 is a waste collectionvehicle. In yet another example, step 2230 may use image classificationalgorithms to analyze the one or more images obtained by step 710 anddetermine whether the second vehicle detected by step 720 is a wastecollection vehicle.

In some embodiments, step 2240 may comprise, for example in response toa determination that the second vehicle detected by step 720 is a wastecollection vehicle (e.g., as determined by step 2230), causing the firstvehicle to initiate an action responding to the second vehicle detectedby step 720. Some non-limiting examples of such action may includesignaling, stopping, changing a speed of the first vehicle, changing amotion direction of the first vehicle, passing the second vehicle,forgoing passing the second vehicle, keeping a minimal distance of atleast a selected length from the second vehicle (for example, where theselected distance is at least 20 feet, at least 50 feet, at least 100feet, at least 190 feet, at least 200 feet, etc.), turning, performing aU-turn, driving in reverse, generating an audible warning, and so forth.For example, step 2240 may transmit a signal to an external device (suchas a device controlling the first vehicle, a device navigating the firstvehicle, the first vehicle, a system within the first vehicle, etc.),and the signal may be configured to cause the external device to causethe first vehicle to initiate the action responding to the secondvehicle detected by step 720. In another example, step 2240 may provideinformation related to the second vehicle detected by step 720 (such asposition, motion, type, activity status, dimensions, etc.) to suchexternal device, and the information may be configured to cause theexternal device to cause the first vehicle to initiate the actionresponding to the second vehicle detected by step 720. In one example,in response to a determination that the second vehicle detected by step720 is not a waste collection vehicle (e.g., as determined by step2230), step 2240 may cause the first vehicle to initiate a secondaction, the second action may differ from the action. In one example, inresponse to a determination that the second vehicle detected by step 720is a waste collection vehicle (e.g., as determined by step 2230), step2240 may cause the first vehicle to initiate a first action, and inresponse to a determination that the second vehicle detected by step 720is not a waste collection vehicle (e.g., as determined by step 2230),step 2240 may cause the first vehicle to initiate a second action, thesecond action may differ from the first action. In another example, inresponse to a determination that the second vehicle detected by step 720is a waste collection vehicle (e.g., as determined by step 2230), step2240 may cause the first vehicle to initiate a first action, and inresponse to a determination that the second vehicle detected by step 720is not a waste collection vehicle (e.g., as determined by step 2230),step 2240 may withhold and/or forgo causing the first vehicle toinitiate the first action.

In some embodiments, the one or more images obtained by step 710 may beanalyzed to determine whether a waste pre-compressor is mounted to thesecond vehicle detected by step 720, for example as described below.Further, in some examples, in response to a determination that a wastepre-compressor is mounted to the second vehicle, step 2230 may determinethat the second vehicle detected by step 720 is a waste collectionvehicle. Further, in some examples, in response to a determination thatno waste pre-compressor is mounted to the second vehicle detected bystep 720, step 2230 may determine that the second vehicle detected bystep 720 is not a waste collection vehicle.

In some examples, the one or more images obtained by step 710 may beanalyzed to determine whether a waste pre-compressor is mounted to thesecond vehicle detected by step 720. For example, a machine learningmodel may be trained using training examples to determine whethervehicles are mounted to waste pre-compressors, and the trained machinelearning model may be used to analyze the one or more images obtained bystep 710 and determine whether a waste pre-compressor is mounted to thesecond vehicle detected by step 720. An example of such training examplemay include an image and/or a video depicting a vehicle, together with alabel indicating whether the depicted vehicle is mounted to a wastepre-compressor. In another example, an artificial neural network (suchas a deep neural networks, convolutional neural networks, etc.) may beconfigured to determine whether vehicles are mounted to wastepre-compressors from images and/or videos, and the artificial neuralnetwork may be used to analyze the one or more images obtained by step710 and determine whether a waste pre-compressor is mounted to thesecond vehicle detected by step 720. In yet another example, objectdetection algorithms (such as visual waste pre-compressor detectors) maybe used to detect waste pre-compressors in the one or more imagesobtained by step 710, the location of the detected waste pre-compressorsmay be compared to the location of the second vehicle detected by step720, and when a location of a detected waste pre-compressor is adjunctto the location of the second vehicle detected by step 720, it may bedetermined that the waste pre-compressor is mounted to the secondvehicle.

In some embodiments, the one or more images obtained by step 710 may beanalyzed to detect waste collector person in a vicinity of the secondvehicle detected by step 720. Further, in some examples, in response toa determination that a waste collector person is in a vicinity of thesecond vehicle detected by step 720, step 2230 may determine that thesecond vehicle detected by step 720 is a waste collection vehicle, andin response to a determination that no waste collector person is in avicinity of the second vehicle detected by step 720, step 2230 maydetermine that the second vehicle detected by step 720 is not a wastecollection vehicle.

In some examples, the one or more images obtained by step 710 may beanalyzed to detect waste collector person in a vicinity of the secondvehicle detected by step 720. For example, a machine learning model maybe trained using training examples to detect waste collector people inimages and/or videos, and the trained machine learning model may be usedto analyze the one or more images obtained by step 710 and detect awaste collector person in a vicinity of the second vehicle detected bystep 720. An example of such training example may include an imageand/or a video, together with a label indicating whether a wastecollector person appears in a particular area of the image and/or of thevideo. In another example, an artificial neural network (such as a deepneural networks, convolutional neural networks, etc.) may be configuredto detect waste collector people in images and/or videos, and theartificial neural network may be used to analyze the one or more imagesobtained by step 710 and detect a waste collector person in a vicinityof the second vehicle detected by step 720. In yet another example, aperson detection algorithm may be used to detect waste collector personin a vicinity of the second vehicle detected by step 720 in the one ormore images obtained by step 710.

In some embodiments, a state of the waste collection vehicle may bedetermined, for example as described below. Some non-limiting examplesof such state may include engaged in waste collection, not engaged inwaste collection, and so forth. In some examples, in response to a firstdetermined state of the waste collection vehicle (such as engaged inwaste collection, not engaged in waste collection, etc.), step 2240 maycause the first vehicle to initiate a first action responding to thesecond vehicle detected by step 720, and in response to a seconddetermined state of the waste collection vehicle (such as engaged inwaste collection, not engaged in waste collection, etc.), step 2240 maywithhold and/or forgo causing the first vehicle to initiate the firstaction. In some examples, in response to a first determined state of thewaste collection vehicle (such as engaged in waste collection, notengaged in waste collection, etc.), step 2240 may cause the firstvehicle to initiate a first action responding to the second vehicledetected by step 720, and in response to a second determined state ofthe waste collection vehicle (such as engaged in waste collection, notengaged in waste collection, etc.), step 2240 may cause the firstvehicle to initiate a second action responding to the second vehicledetected by step 720, where the second action may differ from the firstaction.

In some examples, the one or more images obtained by step 710 may beanalyzed to determine a state of the waste collection vehicle. Somenon-limiting examples of such state may include engaged in wastecollection, not engaged in waste collection, and so forth. For example,a machine learning model may be trained using training examples todetermine states of waste collection vehicles by analyzing images and/orvideos, and the trained machine learning model may be used to analyzethe one or more images obtained by step 710 and determine the state ofthe waste collection vehicle. An example of such training example mayinclude an image and/or a video depicting a waste collection vehicle,together with a label indicating the state of the depicted wastecollection vehicle. In another example, an artificial neural network(such as a deep neural networks, convolutional neural networks, etc.)may be configured to determine states of waste collection vehicles byanalyzing images and/or videos, and the artificial neural network may beused to analyze the one or more images obtained by step 710 anddetermine the state of the waste collection vehicle.

In some examples, audio data captured using one or more audio sensorsfrom an environment of the first vehicle may be obtained and/or analyzedto determine a state of the waste collection vehicle. Some non-limitingexamples of such state may include engaged in waste collection, notengaged in waste collection, and so forth. For example, a machinelearning model may be trained using training examples to determinestates of waste collection vehicles by analyzing audio inputs, and thetrained machine learning model may be used to analyze the captured audiodata and determine the state of the waste collection vehicle. An exampleof such training example may include an audio recording of a wastecollection vehicle, together with a label indicating the state of thewaste collection vehicle. In another example, an artificial neuralnetwork (such as a recurrent neural networks, long short-term memoryneural networks, etc.) may be configured to determine states of wastecollection vehicles by analyzing audio inputs, and the artificial neuralnetwork may be used to analyze the captured audio data and determine thestate of the waste collection vehicle. In yet another example, signalprocessing algorithms may be used to analyze the captured audio data anddetermine whether the waste collection vehicle is engaged in wastecollection or not.

In some examples, a motion of the second vehicle may be determined, forexample as described above. Further, in some examples, in response tothe determined motion of the second vehicle and/or the determinationthat the second vehicle is a waste collection vehicle and/or based onthe determined state of the waste collection vehicle, the first vehiclemay be caused to initiate the action (for example as described above).For example, in response to a first determined motion of the secondvehicle and the determination that the second vehicle is a wastecollection vehicle, the first vehicle may be caused to initiate a firstaction; in response to a second determined motion of the second vehicleand the determination that the second vehicle is a waste collectionvehicle, the first vehicle may be caused to initiate a second action;and in response to the first determined motion of the second vehicle andthe determination that the second vehicle is not a waste collectionvehicle, the first vehicle may be caused to initiate a third action. Inanother example, in response to a first determined motion of the secondvehicle and the determination that the second vehicle is a wastecollection vehicle, the first vehicle may be caused to initiate a firstaction; and in response to a second determined motion of the secondvehicle and the determination that the second vehicle is a wastecollection vehicle, causing the first vehicle to initiate the firstaction may be withheld and/or forwent. For example, in response to afirst determined motion of the second vehicle and a first determinedstate of the waste collection vehicle, the first vehicle may be causedto initiate a first action; in response to a second determined motion ofthe second vehicle and the first determined state of the wastecollection vehicle, the first vehicle may be caused to initiate a secondaction; and in response to the first determined motion of the secondvehicle and a second determined state of the waste collection vehicle,the first vehicle may be caused to initiate a third action. In anotherexample, in response to a first determined motion of the second vehicleand a first determined state of the waste collection vehicle, the firstvehicle may be caused to initiate a first action; in response to asecond determined motion of the second vehicle and the first determinedstate of the waste collection vehicle and/or in response to the firstdetermined motion of the second vehicle and a second determined state ofthe waste collection vehicle, causing the first vehicle to initiate thefirst action may be withheld and/or forwent.

In some examples, it may be determining that the second vehicle issignaling, for example as described above. Further, in some examples, atype of the signaling of the second vehicle may be determined, forexample as described above. Some non-limiting examples of such type ofsignaling may include signaling using light, signaling using sound,signaling left, signaling right, signaling break, signaling hazardwarning, signaling reversing, signaling warning, and so forth. Further,in some examples, in response to the determination that the secondvehicle is signaling and/or the determined type of the signaling and/orthe determination that the second vehicle is a waste collection vehicleand/or based on the determined state of the waste collection vehicle,the first vehicle may be caused to initiate the action (for example asdescribed above). For example, in response to a first determined type ofthe signaling and the determination that the second vehicle is a wastecollection vehicle, the first vehicle may be caused to initiate a firstaction; in response to a second determined type of the signaling and thedetermination that the second vehicle is a waste collection vehicle, thefirst vehicle may be caused to initiate a second action; and in responseto the first determined type of the signaling and the determination thatthe second vehicle is not a waste collection vehicle, the first vehiclemay be caused to initiate a third action. In another example, inresponse to a first determined type of the signaling and thedetermination that the second vehicle is a waste collection vehicle, thefirst vehicle may be caused to initiate a first action; in response to asecond determined type of the signaling and the determination that thesecond vehicle is a waste collection vehicle and/or in response to thefirst determined type of the signaling and the determination that thesecond vehicle is not a waste collection vehicle, causing the firstvehicle to initiate the first action may be withheld and/or forwent. Forexample, in response to a first determined type of the signaling and afirst determined state of the waste collection vehicle, the firstvehicle may be caused to initiate a first action; in response to asecond determined type of the signaling and the first determined stateof the waste collection vehicle, the first vehicle may be caused toinitiate a second action; and in response to the first determined typeof the signaling and a second determined state of the waste collectionvehicle, the first vehicle may be caused to initiate a third action. Inanother example, in response to a first determined type of the signalingand a first determined state of the waste collection vehicle, the firstvehicle may be caused to initiate a first action; in response to asecond determined type of the signaling and the first determined stateof the waste collection vehicle and/or in response to the firstdetermined type of the signaling and a second determined state of thewaste collection vehicle, causing the first vehicle to initiate thefirst action may be withheld and/or forwent. In an additional example,in response to the determination that the second vehicle is signalingand the determination that the second vehicle is a waste collectionvehicle, the first vehicle may be caused to initiate a first action; andin response to the determination that the second vehicle is notsignaling and the determination that the second vehicle is a wastecollection vehicle, the first vehicle may be caused to initiate a secondaction. In yet another example, in response to the determination thatthe second vehicle is signaling and the determination that the secondvehicle is a waste collection vehicle, the first vehicle may be causedto initiate a first action; and in response to the determination thatthe second vehicle is not signaling and the determination that thesecond vehicle is a waste collection vehicle, causing the first vehicleto initiate the first action may be withheld and/or forwent. In anadditional example, in response to the determination that the secondvehicle is signaling and a first determined state of the wastecollection vehicle, the first vehicle may be caused to initiate a firstaction; and in response to the determination that the second vehicle isnot signaling and the first determined state of the waste collectionvehicle, the first vehicle may be caused to initiate a second action. Inyet another example, in response to the determination that the secondvehicle is signaling and a first determined state of the wastecollection vehicle, the first vehicle may be caused to initiate a firstaction; and in response to the determination that the second vehicle isnot signaling and the first determined state of the waste collectionvehicle, causing the first vehicle to initiate the first action may bewithheld and/or forwent.

In some examples, a position of the second vehicle may be determined,for example as described above. For example, the determined position maybe in relation to the ground, in relation to a map, in relation to aroad, in relation to a lane, in relation to an object in theenvironment, in relation to the at least part of the first vehicle, inrelation to the image sensor, and so forth. Further, in some examples,it may be determining that the second vehicle is in a lane of the firstvehicle, for example as described above. Further, in some examples, itmay be determining that the second vehicle is in a planned path of thefirst vehicle, for example as described above. Further, in someexamples, in response to the determined position of the second vehicleand/or the determination that the second vehicle is a waste collectionvehicle and/or based the first determined state of the waste collectionvehicle, the first vehicle may be caused to initiate the action (forexample as described above). For example, in response to a firstdetermined position of the second vehicle and the determination that thesecond vehicle is a waste collection vehicle, the first vehicle may becaused to initiate a first action; in response to a second determinedposition of the second vehicle and the determination that the secondvehicle is a waste collection vehicle, the first vehicle may be causedto initiate a second action; and in response to the first determinedposition of the second vehicle and the determination that the secondvehicle is not a waste collection vehicle, the first vehicle may becaused to initiate a third action. In another example, in response to afirst determined position of the second vehicle and the determinationthat the second vehicle is a waste collection vehicle, the first vehiclemay be caused to initiate a first action; in response to a seconddetermined position of the second vehicle and the determination that thesecond vehicle is a waste collection vehicle and/or in response to thefirst determined position of the second vehicle and the determinationthat the second vehicle is not a waste collection vehicle, causing thefirst vehicle to initiate the first action may be withheld and/orforwent. In an additional example, in response to the determination thatthe second vehicle is in a lane and/or a planned path of the firstvehicle and the determination that the second vehicle is a wastecollection vehicle, the first vehicle may be caused to initiate a firstaction; and in response to the determination that the second vehicle isnot in a lane and/or is not in a planned path of the first vehicle andthe determination that the second vehicle is a waste collection vehicle,the first vehicle may be caused to initiate a second action. In yetanother example, in response to the determination that the secondvehicle is in a lane and/or a planned path of the first vehicle and thedetermination that the second vehicle is a waste collection vehicle, thefirst vehicle may be caused to initiate a first action; and in responseto the determination that the second vehicle is not in a lane and/or isnot in a planned path of the first vehicle and the determination thatthe second vehicle is a waste collection vehicle, causing the firstvehicle to initiate the first action may be withheld and/or forwent. Forexample, in response to a first determined position of the secondvehicle and a first determined state of the waste collection vehicle,the first vehicle may be caused to initiate a first action; in responseto a second determined position of the second vehicle and the firstdetermined state of the waste collection vehicle, the first vehicle maybe caused to initiate a second action; and in response to the firstdetermined position of the second vehicle and a second determined stateof the waste collection vehicle, the first vehicle may be caused toinitiate a third action. In another example, in response to a firstdetermined position of the second vehicle and a first determined stateof the waste collection vehicle, the first vehicle may be caused toinitiate a first action; in response to a second determined position ofthe second vehicle and the first determined state of the wastecollection vehicle and/or in response to the first determined positionof the second vehicle and a second determined state of the wastecollection vehicle, causing the first vehicle to initiate the firstaction may be withheld and/or forwent. In an additional example, inresponse to the determination that the second vehicle is in a laneand/or a planned path of the first vehicle and a first determined stateof the waste collection vehicle, the first vehicle may be caused toinitiate a first action; and in response to the determination that thesecond vehicle is not in a lane and/or is not in a planned path of thefirst vehicle and the first determined state of the waste collectionvehicle, the first vehicle may be caused to initiate a second action. Inyet another example, in response to the determination that the secondvehicle is in a lane and/or a planned path of the first vehicle and afirst determined state of the waste collection vehicle, the firstvehicle may be caused to initiate a first action; and in response to thedetermination that the second vehicle is not in a lane and/or is not ina planned path of the first vehicle and the first determined state ofthe waste collection vehicle, causing the first vehicle to initiate thefirst action may be withheld and/or forwent.

In some embodiments, systems, methods and computer readable media forcontrolling vehicles in response to windows are provided. One challengeof autonomous driving is to identify stopping positions for theautonomous vehicle. The provided systems, methods and computer readablemedia for controlling vehicles may detect a window in the environment(such as a drive-through window, a window of another vehicle, agatekeeper window, etc.), and select the stopping position based on thedetected window.

FIG. 23 illustrates an example of a method 2300 for controlling vehiclesin response to windows. In this example, method 2300 may comprise:obtaining images captured from an environment of a vehicle (Step 710);analyzing the images to detect a first window in the environment (Step2320); and navigating the vehicle to a stopping position (Step 2330),where in the stopping position a window of the vehicle is positioned ata selected position with respect to the first window. In someimplementations, method 2300 may comprise one or more additional steps,while some of the steps listed above may be modified or excluded. Insome implementations, one or more steps illustrated in FIG. 23 may beexecuted in a different order and/or one or more groups of steps may beexecuted simultaneously and/or a plurality of steps may be combined intosingle step and/or a single step may be broken down to a plurality ofsteps. Some non-limiting examples of such first window may include awindow of a building, a window of a second vehicle, a window configuredto allow a service provider (such as a sales person, a gatekeeper, etc.)to communicate with a passenger of the vehicle, a window configured toallow a service provider to exchange items (such as payment methods,products, etc.) with a passenger of the vehicle, a drive-through window,allow a service provider to deliver an item to a passenger of thevehicle, allow a service provider to receive an item to a passenger ofthe vehicle, and so forth.

In some embodiments, step 2320 may comprise analyzing the one or moreimages obtained by step 710 to detect a first window in the environment.For example, step 2320 may use an object detection algorithm to detectthe window. In another example, a machine learning model may be trainedusing training examples to detect windows in images and/or videos, andstep 2320 may use the trained machine learning model to analyze the oneor more images obtained by step 710 to detect the first window. Anexample of such training example may include an image and/or a video,together with a label indicating a depiction of a window in the imageand/or in the video, or together with a label indicating that no windowin depicted in the image and/or in the video. In yet another example, anartificial neural network (such as a deep neural networks, convolutionalneural networks, etc.) may be configured to detect windows in imagesand/or videos, and step 2320 may use the artificial neural network toanalyze the one or more images obtained by step 710 to detect the firstwindow.

In some embodiments, step 2330 may comprise navigating the vehicle to astopping position, where in the stopping position a window of thevehicle is positioned at a selected position with respect to the firstwindow. For example, at the stopping position the window of the vehiclemay be positioned less than a selected distance from the first window(for example, less than two meters, less than one meter, than halfmeter, less than one decimeter, and so forth). In another example, atthe stopping position the window of the vehicle may be positionedsubstantially parallel to the first window (for example, where thedegree between the window of the vehicle and the first window is at mostone degree, is at most two degrees, is at most five degrees, is at mostten degrees, is at most twenty degrees, and so forth). In some examples,step 2330 may select the stopping position (such as a real worldposition, a relative position with respect to the window, a relativeposition with respect to the current position of the vehicle, a set ofcoordinates, an orientation, a direction, etc.) based on properties ofthe first window. Some non-limiting examples of such properties of thefirst window may include absolute real world position of the window,relative real world position of the window (for example, with respect tothe vehicle, with respect to the ground, etc.), position of the windowin the one or more images, type of window, state of the window (such as‘close’, ‘open’, ‘partly close’, ‘fully open’, ‘active’, ‘operative’,‘occupied’, ‘idle’, ‘out of service’, ‘inoperative’, ‘unoccupied’,etc.), height of the window (for example, with respect to ground,absolute, etc.), dimensions of the window (such as width, height, area,etc.), and so forth. For example, such properties of the first windowmay be determined by analyzing the one or more images, for example usinga classification model, using an object detection algorithm, and soforth. In another example, the first window (or an external systemassociated with the first window) may be configured to transmit a signalindicating one or more such properties of the first window, and theproperties of the first window may be determined by receiving and/oranalyzing the transmitted signal. In one example, in response to a firstdetermined property of the first window, step 2330 may select a firststopping position, and in response to a second determined property ofthe first window, step 2330 may select a second stopping position, thesecond stopping position may differ from the first stopping position. Insome examples, step 2330 may transmit a signal to an external device(such as a device controlling the vehicle, a device navigating thevehicle, the vehicle, a system within the vehicle, etc.), and the signalmay be configured to cause the external device to cause the vehicle tomove to the stopping position. In another example, information relatedto the stopping position and/or the first window may be provided to suchexternal device, and the information may cause the external device tocause the vehicle to move to the stopping position.

In some embodiments, method 2300 may further comprise causing the windowof the vehicle to open, for example after step 2330, before step 2330,concurrently with step 2330, and so forth. For example, a signal may betransmitted to an external device (such as a device controlling thewindow of the vehicle, a device controlling the vehicle, the vehicle, asystem within the vehicle, etc.), and the signal may be configured tocause the external device to cause the window of the vehicle to open.

In some examples, method 2300 may further comprise analyzing the one ormore images obtained by step 710 to determine a state of the firstwindow, for example as described below. In some other examples, thefirst window (or an external system associated with the first window)may be configured to transmit a signal indicating the state of the firstwindow, and the state of the first window may be determined by receivingand/or analyzing the transmitted signal. Further, in some examples, inresponse to a first determined state of the first window, step 2330 maynavigate the vehicle to the stopping position, and in response to asecond determined state of the first window, step 2330 may forgonavigating the vehicle to the stopping position. Some non-limitingexamples of such first determined state may include ‘open’, ‘active’,‘operative’, ‘occupied’, and so forth. Some non-limiting examples ofsuch second determined state of the first window may include ‘close’,‘out of service’, ‘inoperative’, ‘unoccupied’, and so forth.

In some examples, the one or more images obtained by step 710 may beanalyzed to determine a state of the first window. Some non-limitingexamples of such state may include ‘close’, ‘open’, ‘partly close’,‘fully open’, ‘active’, ‘operative’, ‘occupied’, ‘idle’, ‘out ofservice’, ‘inoperative’, ‘unoccupied’, and so forth. For example, amachine learning model may be trained using training examples todetermine states of windows from images and/or videos, and the trainedmachine learning model may be used to analyze the one or more imagesobtained by step 710 to determine a state of the first window. Anexample of such training examples may include an image and/or a videodepicting a window, together with a label indicating the state of thedepicted window. In one example, a visual indicator (such as a sign, a‘Sorry We're Closed’ sign, an ‘Open’ sign, a led indicator, a visualtextual indicator, etc.) may indicate the state of the first window, theanalysis of the one or more images obtained by step 710 may identify thevisual indicator and determine the state of the first window based onthe identified visual indicator. In another example, the analysis of theone or more images obtained by step 710 may identify whether a personoccupies a station corresponding to the first window (for example, usinga face detector, using an image classification algorithm, etc.) anddetermine the state of the first window based on whether a personoccupies the station.

In some embodiments, step 2330 may select the selected position withrespect to the first window based on an analysis of the one or moreimages obtained by Step 710. For example, a machine learning model maybe trained using training examples to select position with respect towindows from images and/or videos, and step 2330 may use the trainedmachine learning model to analyze the one or more images obtained byStep 710 and select the selected position with respect to the firstwindow. An example of such training example may include a sample imageof a sample window, together with a label indicating a desired selectionof position with respect to the sample window. In one example, the oneor more images obtained by Step 710 may be analyzed to determine a typeof the first window (for example, using an image classifier trained toclassify windows to different type using training examples), and step2330 may select the selected position with respect to the first windowbased on the determined type of the first window. Some non-limitingexamples of such type may include service window, casement, awming,manned, unmanned, openable, open, close, a drive-through window, awindow of another vehicle, a gatekeeper window, and so forth. In anotherexample, the one or more images obtained by Step 710 may be analyzed toestimate a height of the first window (for example, by analyzing depthimage to directly measure the height from the image, by estimating theheight from a 2D image using a model configured to estimate 3D positionsfrom 2D images, and so forth), and step 2330 may select the selectedposition with respect to the first window based on the estimated heightof the first window. The height may be estimated relative to a ground,relative to another object, relative to a road, relative to the vehicle,absolute, and so forth. In yet another example, the one or more imagesmay be analyzed to determine road conditions in a vicinity of the firstwindow, and step 2330 may select the selected position with respect tothe first window based on the determined road conditions in the vicinityof the first window. Some non-limiting examples of such road conditionsmay include dry, wet, icy, snowy, cracked, blocked, partly blocked, andso forth. For example, a machine learning model may be trained usingtraining examples to determine road conditions in an area from images,and step 2330 may use the trained machine learning model to analyze theone or more images and determine road conditions in a vicinity of thefirst window. An example of such training example may include a sampleimage of a sample road with an indication of an area, together with alabel indicating a condition of the sample road.

In some embodiments, the first window may be configured to allow aservice provider to exchange items with a passenger of the vehicle.Further, in some examples, method 2300 may comprise obtaining one ormore additional images captured from the environment, and analyzing theone or more additional images to determine a transfer of at least oneitem between the service provider and the passenger of the vehiclethrough the first window. For example, method 2300 may analyze the oneor more additional images to determine a transfer of items of particulartypes, of items of particular dimensions, of items of any type, and soforth. For example, the one or more additional images may be capturedafter step 2330, any may be obtained using step 710. For example, theone or more additional images may be captured using at least one imagesensor included in the one or more image sensors used to capture the oneor more images of method 2300, using at least one image sensor notincluded in the one or more image sensors used to capture the one ormore images of method 2300, and so forth. In one example, a machinelearning model may be trained using training examples to determine thatitems are transferred between service providers and passengers ofvehicles through windows (for example, of items of particular types, ofitems of particular dimensions, of items of any type, etc.), and thetrained machine learning model may be used to analyze the additional oneor more images and determine the transfer of the at least one itembetween the service provider and the passenger of the vehicle throughthe first window. An example of such training example may include animage and/or a video, together with a label indicating whether the imageand/or the video depicts a transfer of an item (or of an item of aparticular type, an item of particular dimension, etc.) between aservice providers and a passenger of a vehicle through a window. In oneexample, an artificial neural network (such as a deep neural networks,convolutional neural networks, etc.) may be configured to determine thatitems are transferred between service providers and passengers ofvehicles through windows (for example, of items of particular types, ofitems of particular dimensions, of items of any type, etc.), and theartificial neural network may be used to analyze the additional one ormore images and determine the transfer of the at least one item betweenthe service provider and the passenger of the vehicle through the firstwindow.

In some examples, method 2300 may further comprise causing an actionand/or providing information in response to a determination of thetransfer of the at least one item (or of an item of a particular type,an item of particular dimension, etc.) between the service provider andthe passenger of the vehicle through the first window. Some non-limitingexamples of such actions may include navigating the vehicle (forexample, from the stopping position, to a selected position, in aselected direction, etc.), moving the vehicle, closing the first window,recording the transfer of the at least one item, updating billingrecords according to the transfer of the at least one item, and soforth.

In some embodiments, the first window may be configured to allow aservice provider to communicate with a passenger of the vehicle.Further, in some examples, method 2300 may further comprise obtainingaudio data captured using one or more audio sensors, the obtained audiodata may include a communication between the service provider and thepassenger of the vehicle through the first window, and analyzing thecommunication between the service provider and the passenger of thevehicle. For example, the obtained audio data may be analyzed usingspeech recognition algorithms to determine the content of thecommunication between the service provider and the passenger of thevehicle, and the content of the communication may be analyzed, forexample using a Natural Language Processing algorithms. In anotherexample, the obtained audio data may be analyzed using speakerrecognition algorithms to determine an amount of communication betweenthe service provider and the passenger of the vehicle (or an amount ofspeech produced by the passenger in the communication, an amount ofspeech produced by the service provider in the communication, and soforth). In some examples, the obtained audio data may be analyzed todetermine an ending of communication between the service provider andthe passenger of the vehicle. For example, a machine learning model maybe trained using training examples to identify endings of conversationsin audio streams, and the trained machine learning model may be used toanalyze the obtained audio data and determine the ending ofcommunication between the service provider and the passenger of thevehicle. An example of such training example may include an audiorecording, together with a label indicating an ending of a conversationwithin the audio recording. In another example, an artificial neuralnetwork (such as a recurrent neural networks, long short-term memoryneural networks, etc.) may be configured to identify endings ofconversations in audio streams, and the artificial neural network may beused to analyze the obtained audio data and determine the ending ofcommunication between the service provider and the passenger of thevehicle.

In one example, method 2300 may further comprise causing an actionand/or providing information based on the analysis of the communicationbetween the service provider and the passenger of the vehicle. In oneexample, method 2300 may further comprise causing the first window toclose based on the analysis of the communication between the serviceprovider and the passenger of the vehicle, for example in response to adetermination of the ending of communication between the serviceprovider and the passenger of the vehicle. In another example, method2300 may further comprise causing the first vehicle to move based on theanalysis of the communication between the service provider and thepassenger of the vehicle. For example, method 2300 may cause the firstvehicle to move to a position selected based on the analysis of thecommunication between the service provider and the passenger of thevehicle, to a direction selected based on the analysis of thecommunication between the service provider and the passenger of thevehicle, and so forth.

It will also be understood that the system according to the inventionmay be a suitably programmed computer, the computer including at least aprocessing unit and a memory unit. For example, the computer program canbe loaded onto the memory unit and can be executed by the processingunit. Likewise, the invention contemplates a computer program beingreadable by a computer for executing the method of the invention. Theinvention further contemplates a machine-readable memory tangiblyembodying a program of instructions executable by the machine forexecuting the method of the invention.

What is claimed is:
 1. A non-transitory computer readable medium storinga software program comprising data and computer implementableinstructions for carrying out a method for controlling vehicles inresponse to windows, the method comprising: obtaining one or more imagescaptured using one or more image sensors from an environment of avehicle; analyzing the one or more images to detect a first window inthe environment; and navigating the vehicle to a stopping position,wherein in the stopping position a window of the vehicle is positionedat a selected position with respect to the first window, and wherein inthe stopping position the window of the vehicle is positionedsubstantially parallel to the first window.
 2. The non-transitorycomputer readable medium of claim 1, wherein the method furthercomprises causing the window of the vehicle to open.
 3. Thenon-transitory computer readable medium of claim 1, wherein the firstwindow is a window of a building.
 4. The non-transitory computerreadable medium of claim 1, wherein the first window is a window of asecond vehicle.
 5. The non-transitory computer readable medium of claim1, wherein the first window is configured to allow a service provider toexchange items with a passenger of the vehicle.
 6. The non-transitorycomputer readable medium of claim 5, wherein the method furthercomprises: obtaining one or more additional images captured from theenvironment; and analyzing the one or more additional images todetermine a transfer of at least one item between the service providerand the passenger of the vehicle through the first window.
 7. Thenon-transitory computer readable medium of claim 6, wherein the methodfurther comprises causing an action in response to the determination ofthe transfer of the at least one item between the service provider andthe passenger of the vehicle through the first window.
 8. Thenon-transitory computer readable medium of claim 1, wherein the firstwindow is configured to allow a service provider to communicate with apassenger of the vehicle.
 9. The non-transitory computer readable mediumof claim 8, wherein the method further comprises: obtaining audio datacaptured using one or more audio sensors and including a communicationbetween the service provider and the passenger of the vehicle throughthe first window; and analyzing the communication between the serviceprovider and the passenger of the vehicle.
 10. The non-transitorycomputer readable medium of claim 9, wherein the method furthercomprises causing the first window to close based on the analysis of thecommunication between the service provider and the passenger of thevehicle.
 11. The non-transitory computer readable medium of claim 9,wherein the method further comprises causing the first vehicle to movebased on the analysis of the communication between the service providerand the passenger of the vehicle.
 12. The non-transitory computerreadable medium of claim 1, wherein, at the stopping position, thewindow of the vehicle is positioned less than one meter from the firstwindow.
 13. The non-transitory computer readable medium of claim 1,wherein, at the stopping position, the window of the vehicle ispositioned less than half meter from the first window.
 14. Thenon-transitory computer readable medium of claim 1, wherein the methodfurther comprises: analyzing the one or more images to determine a stateof the first window; and when the determined state of the first windowis open, navigating the vehicle to the stopping position.
 15. Thenon-transitory computer readable medium of claim 1, wherein the methodfurther comprises: analyzing the one or more images to determine a typeof the first window; and selecting the selected position with respect tothe first window based on the determined type of the first window. 16.The non-transitory computer readable medium of claim 1, wherein themethod further comprises: analyzing the one or more images to estimate aheight of the first window; and selecting the selected position withrespect to the first window based on the estimated height of the firstwindow.
 17. The non-transitory computer readable medium of claim 1,wherein the method further comprises: analyzing the one or more imagesto determine road conditions in a vicinity of the first window; andselecting the selected position with respect to the first window basedon whether the determined road conditions in the vicinity of the firstwindow is icy.
 18. A system for controlling vehicles in response towindows, the system comprising: at least one processing unit configuredto: obtain one or more images captured using one or more image sensorsfrom an environment of a vehicle; analyze the one or more images todetect a first window in the environment; and navigate the vehicle to astopping position, wherein in the stopping position a window of thevehicle is positioned at a selected position with respect to the firstwindow, and wherein in the stopping position the window of the vehicleis positioned substantially parallel to the first window.
 19. A methodfor controlling vehicles in response to windows, the method comprising:obtaining one or more images captured using one or more image sensorsfrom an environment of a vehicle; analyzing the one or more images todetect a first window in the environment; and navigating the vehicle toa stopping position, wherein in the stopping position a window of thevehicle is positioned at a selected position with respect to the firstwindow, and wherein in the stopping position the window of the vehicleis positioned substantially parallel to the first window.
 20. Thenon-transitory computer readable medium of claim 1, wherein the methodfurther comprises: analyzing the one or more images to determinedimensions of the first window; and selecting the selected position withrespect to the first window based on the determined dimensions of thefirst window.